Directory Structure
In order for the ingest script to properly identify new runs and ingest the available data the following directory structure and json files are expected in the order and format described below.
shots/
index.json
{shot}/
{run}/
time.json
{time_step_1}/
...
{time_step_n}/
variables.json
images.tar.gz/
{plot_category}/
plots.json
{variable_1}.png
...
{variable_n}.png
{plot_category}.bp
File and directory expectations
Shots Directory
The shots directory must be named shots
and is the very top-level directory that the script is looking for. This should only need to be created once and new data will simply be added to it. Contains one or more simulations as well as the index.json file.
Simulation Directory
Contains one or more runs.
index.json
The index file contains a list of objects, where each object describes a simulation directory. This data is used to build the directory in the database and is used to associate runs with the correct simulation. The additonal keys are saved to the metadata. The shot_name
and run_name
keys are required. The username
, machine_name
, and date
are optional. All keys expect string values.
[
{
"shot_name": "my_shot_name",
"run_name": "my_run_name",
"username": "bnmajor",
"machine_name": "crusher",
"date": "2022-10-19T10:13:10+422603"
}
]
Run Directory
Contains a directory for each time step as well as the time.json file.
time.json
This file should be updated throughout the run for live runs. The current
key is an integer that represents the current time step that the run is on. If the run is completed this will be the last time step. The complete
key is a boolean that is used to indicate if the run is complete. The started
key is a boolean that indicates if the run has been started. The current
and complete
keys are required and the started
key is optional.
{
"current": 650,
"complete": true,
"started": true
}
Time Step Directory
Contains an images tarball as well as a variables.json file.
variables.json
The variables file should contain a list of objects, where each object describes a variable that is available for the current time step. The file_name
is a string that describes the path to the plot data inside of the tarball. The attribute_name
is the variable name and should be the key that the plot data is stored under in the bp file. The group_name
is the group (the top-level directory in the tarball) that the plot belongs to. The time
represents the actual time assocaited with this time step. All keys are required.
[
{
"file_name": "heatload-images/650/hplot-heatload/HeatLoad-650.bp",
"attribute_name": "eenflux side=0",
"group_name": "Heat Load",
"time": 0.0001514490625775283
},
{
"file_name": "1D-images/650/visualization-1D/diag-1D.0.151449.bp",
"attribute_name": "Elec. g.c. Density (m^-3)",
"group_name": "diag-1D",
"time": 0.00015144906257752824
},
{
"file_name": "heat-images/650/visualization-heat/diag-heat.0.151449.bp",
"attribute_name": "Heat (MW) [outboard]",
"group_name": "diag-heat",
"time": 0.00015144906257752824
},
{
"file_name": "mesh-images/650/visualization-mesh/diag-mesh.0.151449.bp",
"attribute_name": "Q_DPOT",
"group_name": "diag-mesh",
"time": 0.00015144906257752824
}
]
Images Tarball
Contains one or more group directories.
Groups
Contains an adios (.bp) file, a plots.json file, and png
image files for each variable available at this time step.
plots.json
The plots file will contain the same json as the variables.json file, but it will be limited to the variables that belong to this category/group.
[
{
"file_name": "heatload-images/650/hplot-heatload/HeatLoad-650.bp",
"attribute_name": "eenflux side=0",
"group_name": "Heat Load",
"time": 0.0001514490625775283
}
]
Adios File
Attributes
There should be a json-stringified attribute for each each variable that contains the high-level information about the plot.
Plotly Example
One-dimensional plots are currently generated with Plotly and the type
should indicate the plot type that is expected for this variable. The x
key should be a string and will represent the variable that the x-axis data is stored under in the bp file. The xlable
is also a string, but this is the exact string that will be used to label the x-axis. The y
key should be a list of one or more strings that will represent the variable(s) that the y-axis data is stored under in the bp file. Similarly, the yname
should also be a list the same length as the y
key, with string values to be used when labeling data in the plot legend. The ylabel
is actually the title of the plot.
{
"type": "lines",
"x": "time",
"y": ["enn=0", "enn=1", "enn=2", "enn=3"],
"yname": ["n=0", "n=8", "n=16", "n=24"],
"xlabel": "Time (ms)",
"ylabel": "$\\sqrt{<|\\phi_n/T_0|^2>}$"
}
Mesh Example
Two-dimensional plots are currently generated with VTK. All values for these attribute values are expected to be strings and are used to indicate the variable that the data can be found under in the bp file. Additionally, for these plots the type
tells us not just what type of plot should be produced but also what assumptions we can make about how connectivity data will be provided as well as how to handle this data client-side to produce the expected output.
mesh-colormap
: This plot type is assumed to share the same mesh connectivity across all time steps.colormap
will build the mesh for each new time step. This will also tell the client-side to make some adjustments to the plot generation to produce the expected square plot.
The nodes
key indicates the number of points that are being plotted and the connectivity
value will point to the connectivty data. Much like the 1D plot data the xlabel
will be used to label the x-axis
, the ylabel
will be used to label the y-axis, and the title
will be used to assign the title of the plot. The color
represents the scalar values used to color the plot.
{
"type": "mesh-colormap",
"nodes": "mesh_rz",
"connectivity": "mesh_conn",
"color": "q_apars",
"xlabel": "r (m)",
"ylabel": "z (m)",
"title": "$A_{||} C_s/T_e$ at 0.1514 ms"
}
Colormap Example
{
"type": "colormap",
"x": "psi",
"y": "timeflux",
"color": "i_flux_r_2D",
"xlabel": "Normalized Pol. Flux",
"ylabel": "Time (ms)",
"title": "Radial Heat Flux (MW)"
}
Variables
Variable names are expected to exactly match what is in the attribute string and the data that is used client-side is currently exactly as it is stored in the bp file.
Ingest a Run
Detailed information coming soon
Questions or Comments
For questions about supporting a new plot type, data indestion, or current dashboard issues you can open a new issue.
For questions and discussion around current directory and/or file structure or expectations feel free to start up (or expand on) a topic in our discussions section!