Contents

NEOrg Reference Guide

Special directives

A directive is a general reStructuredText block markup. With the special directives defined by NEOrg, you can fetch data and images, show them effectively, and organize your notes.

Show data in a table - table-data

.. table-data:: path [path ...]

Search data file under the given list of path and show matched data. You can use Unix shell-style pattern matching.

base : string (newlines removed)

This is an optional parameter to specify the directory where the data files are searched from. The data files will be searched from DATADIRPATH if not specified.

For example,

.. table-data:: 2011-02-*/data.json
                2011-03-*/data.json
   :base: my/experiment

will search the files which match to my/experiment/2011-02-*/data.json or my/experiment/2011-03-*/data.json.

file : string (newlines removed)

This is an optional parameter to specify the name of the data file. If the tailing parts of the arguments are the you can specify it using this option.

For example,

.. table-data:: 2011-02-* 2011-03-*
   :file: data.json
   :base: my/experiment

will search the files which matche to my/experiment/2011-02-*/data.json or my/experiment/2011-03-*/data.json.

data : text [, text, ...]
A comma- or space-separated list of the dictionary path. You can use Unix shell-style pattern matching.
widths : integer [, integer...]
A comma- or space-separated list of relative column widths.
link : text [, text]

A comma- or space-separated list of path to the link(s). Magic words %(path)s and %(relpath)s are available.

%(path)s
This represents full path to the parent directory of the data file.
%(relpath)s
This is the relative path of the %(path)s from the base directory.
path-order : {‘sort’, ‘sort_r’}

Sort the matched data paths. sort will sort the paths in alphabetical order and sort_r is the reversed version of sort. The default is 'sort'. Note that

.. table-data:: 2011-02-*/data.json
                2011-03-*/data.json
   :path-order: sort_r

does not makes 2011-03-01/data.json higher than 2011-02-01/data.json. You will need to write

.. table-data:: 2011-03-*/data.json
                2011-02-*/data.json
   :path-order: sort_r
trans : flag
Transpose the table.
ftype-{TYPE} : text [, text]

A comma- or space-separated list of unix shell-style pattern matching to be matched the path of data file where {TYPE} is a file type. Currently supported file types are: pickle, python, yaml and json. This is used for loading data file with unusual extension, e.g.:

.. table-data:: */data.txt
   :ftype-json: *.txt

Show data and images in a table - table-data-and-image

.. table-data-and-image:: path [path ...]

Search data under the given list of path and show matched data and corresponding image(s). You can use Unix shell-style pattern matching.

base : string (newlines removed)
This is an optional parameter to specify the directory where the data files are searched from. See table-data for the details.
file : string (newlines removed)
Specifies the tailing part of the directory where the data files are searched from. See table-data for the details.
data : text [, text, ...]
A comma- or space-separated list of the dictionary path. See table-data for the details.
image : text [, text, ...]
A comma- or space-separated list of path to the images. The path is the relative path from the parent directory of the data file.
widths : integer [, integer...]
A comma- or space-separated list of relative column widths. Note that the first column is data sub-table and the second and the after are the images specified in :image:.
image-{OPTION} : integer:{VAL} [, integer:{VAL} ...]
integer is the index of the image. {VAL} specifies the value of the {OPTION} of the image directive.
link : text [, text]
A comma- or space-separated list of path to the link(s). See table-data for the details.
path-order : {‘sort’, ‘sort_r’}
Sort the matched data paths. See table-data for the details.
sort : text [, text]
A comma- or space-separated list of the dictionary path. The table will be sorted by values of the keys.
ftype-{TYPE} : text [, text]
Load data file with unusual extension. See table-data for the details.

See the difference of data - dictdiff

.. dictdiff:: path [path ...]

Search data under the given list of path and show the difference of the data. You can use Unix shell-style pattern matching.

base : string (newlines removed)
This is an optional parameter to specify the directory where the data files are searched from. See table-data for the details.
file : string (newlines removed)
Specifies the tailing part of the directory where the data files are searched from. See table-data for the details.
link : text [, text]
A comma- or space-separated list of path to the link(s). See table-data for the details.
include : text [, text]
A comma- or space-separated list of regular expression of the dictionary path to include.
exclude : text [, text]
A comma- or space-separated list of regular expression of the dictionary path to exclude.
path-order : {‘sort’, ‘sort_r’}
Sort the matched data paths. See table-data for the details.
trans : flag
Transpose the table.
ftype-{TYPE} : text [, text]
Load data file with unusual extension. See table-data for the details.

Show effects of the parameter change - grid-images

.. grid-images:: path [path ...]

Search data and show the images related to the data on “grid”. You can use Unix shell-style pattern matching.

In experiments, you may change parameters systematically to see what will happen. For example, when you do experiments varying two parameters a=1, 2, 3 and b=0.1, 0.2, 0.3, you will get 9 results. Viewing the results in a linear list is a bad idea: you should see it in a 3x3 matrix like this!:

  b=0.1 b=0.2 b=0.3
a=1 result with a=1 and b=0.1 result with a=1 and b=0.2 result with a=1 and b=0.3
a=2 result with a=2 and b=0.1 result with a=2 and b=0.2 result with a=2 and b=0.3
a=3 result with a=3 and b=0.1 result with a=3 and b=0.2 result with a=3 and b=0.3

The grid is a N-dimensional version of this matrix. To see it in 2D display (because we don’t have N-dim display), NEOrg generates the nested table.

base : string (newlines removed)
The base path for searching data. Real path to be used This is an optional parameter to specify the directory where the data files are searched from. See table-data for the details.
file : string (newlines removed)
Specifies the tailing part of the directory where the data files are searched from. See table-data for the details.
param : text [, text]
A comma- or space-separated list of the dictionary path for the axes of the grid.
image : text [, text]
A comma- or space-separated list of path to the images. The path is the relative path from the parent directory of the data file.
link : text
A URL to specify the link to a page related to the found data file. Note that unlike the link option of table-data, only one link is supported. See table-data for the details.

Find images - find-images

.. find-images:: path [path ...]

Search images under the given list of path and show matched images. You can use Unix shell-style pattern matching.

base : string (newlines removed)
This is an optional parameter to specify the directory where the data files are searched from. See table-data for the details.
file : string (newlines removed)
Specifies the tailing part of the directory where the data files are searched from. See table-data for the details.

List Pages - list-pages

.. list-pages::

Insert list of sub-pages.

Recently Updated Sub-pages - recent-pages

.. recent-pages::

Insert list of recently updated sub-pages.

num : int
Number of pages to show (default is 10).

Template page - _temp_

When you do experiments repeatedly, you may want to see the results in a fixed format. The template page can be used for that purpose.

The page which has _temp_ in its page path is a template page. The template page is used for generating page. If the page path does not exist but the template page which matches to the page path exists, generated page will be shown.

Examples of the template page path:

  1. /my/page/_temp_/
  2. /my/page/_temp_/_temp_/
  3. /my/page/_temp_/images/
  4. /my/page/_temp_/subdata/_temp_/
  5. /my/page/_temp_/_temp_/subdata
  • /my/page/2011-05-21/ matches to (a)
  • /my/page/2011-05-21/some-data/ matches to (b)
  • /my/page/2011-05-21/images/ matches to (c)
  • /my/page/2011-05-21/subdata/000/ matches to (d)
  • /my/page/2011-05-21/000/subdata/ matches to (e)
  • /my/page/2011-05-21/subdata/subdata/ matches to (e)
{{ args[N] }} (where N is an integer)
N-th replacement of the _temp_ in the URL. For example, at the page /my/page/2011-05-21/subdata/000/ in the above example, {{ args[0] }} and {{ args[1] }} will be replaced by 2011-05-21 and 000.
{{ path }}
This will be replaced by the full path to this directory.
{{ relpath }}
This will be replaced by the relative path from the parent page of the leftmost _temp_ page.

Searching Pages

NEOrg has Whoosh powered searching functionality. For the full description of the query language, see The default query language — Whoosh documentation. The following are quick examples:

Query examples
Query Meaning
alpha AND beta  
alpha beta equivalent to alpha AND beta
alpha NOT (beta OR gamma)  
page_path:MyPage use page path in the search query
page_path:(MyPage SubPage) equivalent to page_path:MyPage page_path:SubPage
te?t test* *b?g* use unix shell-style pattern matching
ninja^2 cowboy bear^0.5 importance: ninja = 2*cowboy = 4*bear

Glossary

directive

A directive is one of the reStructuredText block markup in the following shape:

.. directive-name:: argument(s)
   :option: parameter
   :another_option: another_parameter

   some contents

See reStructuredText Directives for more information and the basic directives defined by docutils.

NEOrg defines various special for displaying and organizing data effectively (see Special directives).

page path
The page path is the URL to the page.
unix shell-style pattern matching

Unix shell-style pattern matching supports the following wild-cards.

Pattern Meaning
* matches everything
? matches any single character
[seq] matches any character in seq
[!seq] matches any character not in seq
base
This is the optional parameter for the directives which searches the data files. See table-data for the details.
This is the optional parameter for the directives which generates the links to the other pages. See table-data for the details.
magic words

Magic words are the special words which are replaced by NEOrg.

Magic words %(path)s and %(relpath)s are available in the link parameter.

Magic words %(root)s and %(neorg)s are available in the configuration file for the variables DATADIRPATH and DATABASE.

dictionary path

To specify the value in the nested dictionary, NEOrg uses period-separated keys. For example, the values in the following nested dictionary can be accessed by the “dictionary path” such as 'key1.nestedkey1'.

{'key1': {
     'nestedkey1': 1,
     'nestedkey2': 2
 }
 'key2': 2
}
regular expression

NEOrg uses python regular expression to filtering out/in dictionary path.

See Regular Expression Syntax — Python documentation for more information about the regular expression syntax.

template page
A template page is the page which has at least one _temp_ in its page path. See also Template page - _temp_.
debug mode

In debug mode, python interactive shell will be invoked in your browser when NEOrg clashes. See Debugging Applications – Werkzeug documentation and Debug Mode — Flask documentation for more information.

Warning

If someone can access to the NEOrg running in debug mode, he can do anything your python can do, such as deleting files. Thus, debug mode should not be used on an untrusted network.

Configuration variables

DATADIRPATH

Path to your data directory. Magic words %(root)s and %(neorg)s are available. The string ~ will be replaced by the user’s home directory. Also, the environment variables are available. For example, you can also use $HOME or ${HOME} to specify your home directory. The default is %(root)s.

%(root)s
The directory you specified by neorg init command. If you did not specify, this is the directory where you ran neorg init command.
%(neorg)s
This is equivalent to %(root)s/.neorg/.
DATABASE

The path to the sqlite data base file. Magic words %(root)s and %(neorg)s, special character ~, and the environment variables are available. The default is '%(neorg)s/neorg.db'.

DEBUG

If it is set to True, neorg serve runs in debug mode always.

FAQs

How do I make a new page?

Just type a page path to the browser’s address bar, e.g.:

http://localhost:8000/my/new/page/

and then you will see the edit form, if the page does not exist.

Other contents

Example usages of the special directives

File tree

The following examples assume we have the file tree under the DATADIRPATH like this:

DATADIRPATH
`- my/
   `- experiment/
      |- 2011-02-25-182001/
      |  |- graph_1.png
      |  |- graph_2.png
      |  `- data.json
      |
      |- 2011-02-25-182002/
      |  |- graph_1.png
      |  |- graph_2.png
      |  `- data.json
      |
      |- 2011-02-25-182003/
      |  |- graph_1.png
      |  |- graph_2.png
      |  `- data.json
      |
      |- 2011-02-26-182001/
      |  |- graph_1.png
      |  |- graph_2.png
      |  `- data.json
      |
      `- 2011-02-26-182002/
         |- graph_1.png
         |- graph_2.png
         `- data.json

Example usages of table-data

Basic example
.. table-data:: my/experiment/2011-02-25-*/data.json
   :data: alpha beta result.gamma
   :link: %(path)s

will produce a table like this:

Data found in: my/experiment/2011-02-25-*/data.json
  alpha beta result.gamma link(s)
my/experiment/2011-02-25-182001/data.json 1 0.1 0.01 (link to the page /my/experiment/2011-02-25-182001/)
my/experiment/2011-02-25-182002/data.json 2 0.2 0.02 (link to the page /my/experiment/2011-02-25-182001/)
my/experiment/2011-02-25-182003/data.json 3 0.3 0.03 (link to the page /my/experiment/2011-02-25-182003/)
Use of :base: option
.. table-data:: 2011-02-25-*/data.json
                2011-02-26-*/data.json
   :base: my/experiment
   :data: alpha beta result.gamma
   :link: %(relpath)s

will produce a table like this:

Data found in: my/experiment/{2011-02-25-*/data.json, 2011-02-26-*/data.json}
  alpha beta result.gamma link(s)
2011-02-25-182001/data.json 1 0.1 0.01 (link to the page 2011-02-25-182001/)
2011-02-25-182002/data.json 2 0.2 0.02 (link to the page 2011-02-25-182001/)
2011-02-25-182003/data.json 3 0.3 0.03 (link to the page 2011-02-25-182003/)
2011-02-26-182001/data.json 1 0.2 0.01 (link to the page 2011-02-26-182001/)
2011-02-26-182002/data.json 2 0.1 0.02 (link to the page 2011-02-26-182001/)
Newest results on top
.. table-data:: 2011-02-25-*/data.json
   :path-order: sort_r
   :base: my/experiment
   :data: alpha beta result.gamma
   :link: %(relpath)s

will produce a table like this:

Data found in: my/experiment/2011-02-25-*/data.json
  alpha beta result.gamma link(s)
2011-02-25-182003/data.json 3 0.3 0.03 (link to the page 2011-02-25-182003/)
2011-02-25-182002/data.json 2 0.2 0.02 (link to the page 2011-02-25-182001/)
2011-02-25-182001/data.json 1 0.1 0.01 (link to the page 2011-02-25-182001/)

Example usages of table-data-and-image

Basic example
.. table-data-and-image:: 2011-02-25-*/data.json
   :image: graph_1.png graph_2.png
   :path-order: sort_r
   :base: my/experiment
   :data: alpha beta result.gamma
   :link: %(relpath)s

will produce a table like this:

Data found in: my/experiment/2011-02-25-*/data.json
2011-02-25-182003/data.json
alpha 3
beta 0.3
result.gamma 0.03
  • (link to the page 2011-02-25-182003/)
(image of 2011-02-25-182003/graph_1.png) (image of 2011-02-25-182003/graph_2.png)
2011-02-25-182002/data.json
alpha 2
beta 0.2
result.gamma 0.02
  • (link to the page 2011-02-25-182002/)
(image of 2011-02-25-182002/graph_1.png) (image of 2011-02-25-182002/graph_2.png)
2011-02-25-182001/data.json
alpha 1
beta 0.1
result.gamma 0.01
  • (link to the page 2011-02-25-182001/)
(image of 2011-02-25-182001/graph_1.png) (image of 2011-02-25-182001/graph_2.png)
Scaling image

To scale the image with the table, you can use :image-width: option.

.. table-data-and-image:: 2011-02-25-*/data.json
   :image: graph_1.png
   :image-width: 0:100%
   :path-order: sort_r
   :base: my/experiment
   :data: alpha beta result.gamma
   :link: %(relpath)s

will produce a table like this:

Data found in: my/experiment/2011-02-25-*/data.json
2011-02-25-182003/data.json
alpha 3
beta 0.3
result.gamma 0.03
  • (link to the page 2011-02-25-182003/)

(image of 2011-02-25-182003/graph_1.png)

<- the width varied with the width of the table. ->

2011-02-25-182002/data.json
alpha 2
beta 0.2
result.gamma 0.02
  • (link to the page 2011-02-25-182002/)
(image of 2011-02-25-182002/graph_1.png)
2011-02-25-182001/data.json
alpha 1
beta 0.1
result.gamma 0.01
  • (link to the page 2011-02-25-182001/)
(image of 2011-02-25-182001/graph_1.png)

Example usages of dictdiff

Basic example
.. dictdiff:: 2011-02-25-*/data.json
   :base: my/experiment
   :link: %(relpath)s
Diff of data found in: my/experiment/2011-02-25-*/data.json
  seed alpha beta result.gamma link(s)
my/experiment/2011-02-25-182001/data.json 8906300472 1 0.1 0.01 (link to the page /my/experiment/2011-02-25-182001/)
my/experiment/2011-02-25-182002/data.json 2634932505 2 0.2 0.02 (link to the page /my/experiment/2011-02-25-182001/)
my/experiment/2011-02-25-182003/data.json 2510325972 3 0.3 0.03 (link to the page /my/experiment/2011-02-25-182003/)
Exclude data

If you don’t want to see seed in the data, you can exclude it using the :exclude: option.

.. dictdiff:: 2011-02-25-*/data.json
   :base: my/experiment
   :link: %(relpath)s
   :exclude: seed
Diff of data found in: my/experiment/2011-02-25-*/data.json
  alpha beta result.gamma link(s)
my/experiment/2011-02-25-182001/data.json 1 0.1 0.01 (link to the page /my/experiment/2011-02-25-182001/)
my/experiment/2011-02-25-182002/data.json 2 0.2 0.02 (link to the page /my/experiment/2011-02-25-182001/)
my/experiment/2011-02-25-182003/data.json 3 0.3 0.03 (link to the page /my/experiment/2011-02-25-182003/)

Example usages of grid-images

Basic example
.. grid-images:: 2011-02-25-*[12]/data.json
                 2011-02-26-*/data.json
   :base: my/experiment
   :image: graph_1.png
   :param: alpha beta
Comparing images of data found in: my/experiment{2011-02-25-*[12]/data.json, 2011-02-26-*/data.json}
  beta=0.1 beta=0.2
alpha=1 (image of 2011-02-25-182001/graph_1.png) (image of 2011-02-26-182001/graph_1.png)
alpha=2 (image of 2011-02-26-182002/graph_1.png) (image of 2011-02-25-182002/graph_1.png)

Example usage of find-images

.. find-images:: 2011-02-25-*/graph_1.png
   :base: my/experiment

will produce paragraphs like this:

my/experiment/2011-02-25-182001/graph_1.png

(image of 2011-02-25-182001/graph_1.png)

my/experiment/2011-02-25-182002/graph_1.png

(image of 2011-02-25-182002/graph_1.png)

my/experiment/2011-02-25-182003/graph_1.png

(image of 2011-02-25-182003/graph_1.png)

Example usages of the template page

With the above data tree, you may want to generate the similar page when you access the page /my/experiment/.... For that purpose, you can make a template page /my/experiment/_temp_ with the following contents:

Table data
==========

.. table-data:: data.json
   :base: my/experiment/{{ args[0] }}/
   :data: *
   :trans:
   :widths: 1 9

Images
======

 .. find-images:: *.png
    :base: my/experiment/{{ args[0] }}/

This template page generates the following page when you access to the page /my/experiment/2011-02-25-182001.

Table data
==========

.. table-data:: data.json
   :base: my/experiment/2011-02-25-182001/
   :data: *
   :trans:
   :widths: 1 9

Images
======

 .. find-images:: *.png
    :base: my/experiment/2011-02-25-182001/

Note that {{ args[0] }} is replaced by 2011-02-25-182001.

The above page can also be generated by the following template pages:

.. an example using "relpath"

Table data
==========

.. table-data:: data.json
   :base: my/experiment/{{ relpath }}
   :data: *
   :trans:
   :widths: 1 9

Images
======

 .. find-images:: *.png
    :base: my/experiment/{{ relpath }}

and

.. an example using "path"

Table data
==========

.. table-data:: data.json
   :base: {{ path }}
   :data: *
   :trans:
   :widths: 1 9

Images
======

 .. find-images:: *.png
    :base: {{ path }}

NEOrg commands

NEOrg sub-commands

usage: neorg [-h] {init,serve} ...

NEOrg - Numerical Experiment Organizer

positional arguments:
  {init,serve}
    init        initialize neorg directory
    serve       start stand-alone webserver

optional arguments:
  -h, --help    show this help message and exit

init

usage: neorg init [-h] [dest]

positional arguments:
  dest        root directory to initialize (default: ".")

optional arguments:
  -h, --help  show this help message and exit

serve

usage: neorg serve [-h] [-p PORT] [-R ROOT] [-b] [--debug]

optional arguments:
  -h, --help            show this help message and exit
  -p PORT, --port PORT  port to listen (default: 8000)
  -R ROOT, --root ROOT  root directory (where `.neorg/` exists)
  -b, --browser         open web browser
  --debug               set DEBUG=True to run in debug mode

ChangeLog

v0.0.4

v0.0.3

  • Pages can be searched from the right top search box.
  • table-data-and-image highlights data with different values.
  • list-pages shows sub-pages as a nested list, not as a flat list.
  • Show sub-pages when the accessed page does not exist but at least one sub-page exists.
  • Use reversed page path as a page title.

v0.0.2

  • Check system version before running neorg serve command.
  • If neorg page creation clashes, show the python traceback so that user still can edit the page to workaround the problem.
  • Bug fixes:

v0.0.1

  • The first release.