[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

How-to use a source block output as code for another source block?

From: Gaëtan André
Subject: How-to use a source block output as code for another source block?
Date: Fri, 6 Dec 2019 19:41:39 +0100
User-agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.3.0


Yesterday I asked this question on IRC and the community kindly
answered. I was also asked to send an email to the mailing list
explaining my use case. Thus this email.

So, in the scope of my current project at work I need to see if Neo4J would
be a valid alternative to PostgreSQL to store our data. Were hoping to
have better flexibility and better performance on some requests.
In order to document the steps of my exploration I decided to use
org-mode which I discovered a couple of weeks ago (I use it mainly to
keep a diary of my work days).

The first step in my investigation is to migrate the data from Postgres
to Neo4j via CSV files. Since our schema contains quite a bit of tables
and relations I decided to generate corresponding SQL COPY ... TO ...
requests by first putting variable data in a array, second generating
requests with a python source bloc and then feed it to a SQL bloc. Here
is the result:

#+tblname: rels-data
| computer    | id | house      | house_id        | EQUIPS      | computer   | | house       | id | block       | block_id        | IN          | house       |
| block        | id | city        | city_id          | IN | block       |
| city         | id | country     | country_id       | IN | city         |
| city         | id | city_group   | city_group_id    | BELONGS_TO | city        |

#+name: import-rels
#+begin_src python :var rels=rels-data :results value verbatim
res = ["""COPY (
   SELECT '{0}' || {1} as ":START_ID", '{2}' || {3} as ":END_ID", '{4}' as ":TYPE"
   FROM "{5}"
) TO '/data/rels_{0}_{2}.csv' WITH CSV HEADER;""".format(*row) for row in rels]
return "\n".join(res)

#+name: export-data
#+header: :engine postgresql
#+header: :dbhost localhost
#+header: :dbuser mrwho
#+header: :database whatever
#+header: :noweb no-export
#+begin_src sql

All the magic is in the use of noweb[1] feature.

If you have any questions or remarks don't hesitate.


[1] https://orgmode.org/manual/noweb.html

reply via email to

[Prev in Thread] Current Thread [Next in Thread]