Scientists are not programmers. Repeat that after me: scientists are not programmers. It’s not their fault; it’s just a lack of proper training.  If you are implementing some algorithm given you by a scientist, it’s important to know this and account for it.

Certain algorithms are not direct – most often for some process which is not easily reversible.  For example, I was given the task of implementing a way of finding the Wet-Bulb temperature, given the Dewpoint temperature, the Dry-Bulb temperature, and the Barometric Pressure.  Accompanying this task was some code, written by a scientist, in some form of BASIC.

To accomplish this, they started with an estimate (the DewPoint Temp) and worked forward, using the known equations to convert wet-bulb temp into dewpoint temp, then compared that result to the known dewpoint (Tdew).  If the result was less than the known dewpoint, they added a constant 0.05 degrees to the estimate, and tried again. When the result exceeded the dewpoint, they called it good and returned the latest estimate as the final answer.

Scientists are not programmers. If you asked them about this, they will say that it gets the right answer.  If you ask them how they came to choose 0.05 as the step size, after the blank stare (while they think about it), you will get an answer something like “Well, that’s the tolerance I want”. If you really press them, they will come up with “Well, any smaller and it’ll take too long – any larger and it’ll not be correct enough”, which is exactly true. That step size is somebody’s wild guess.

Being the obsessive speed freak that I am, I figured a better way.  What the scientist didn’t realize, is that you don’t have to have a constant step size.  With a modicum of further effort, you can adjust the step size dynamically, and get to the final answer much more quickly.

Simply start with a relatively large positive step, and do your estimates as before.  Afterward, make a decision – if you haven’t exceeded your target, step again in the same direction.  If you exceeded the target, don’t simply quit and call it good, REDUCE and REVERSE your step size and go again.  Now you’re heading negative. When you go BELOW your target, REDUCE and REVERSE your step size. Repeat this until the absolute value of your step size is below your tolerance.

In certain cases, this will take LONGER, but in the vast majority of cases where a fine tolerance is needed, this will get a more accurate answer in FEWER iterations.

You of course need to check things out and match your particular case. Use an iteration counter. You always want to reduce your step size when you reverse it: a factor of -1 would never converge and a factor of near -1 would converge slowly. But a factor of -0.01 would reduce your tolerance.  Best to use a factor of -0.1 to -0.4.

I have seen reductions of 100:1 in iteration counts between the original method and this improved search.  In cases where it was worse, it was around 15 vs. 10 iterations, in cases where it was better it was around  30 vs. 500 iterations.

Use your common sense, and don’t hold it against them.  Scientists are not programmers.

If you read the post about en masse operations, you might remember that I pointed out that you should know what is happening behind the scenes. Here is a particular case where what looks like simpler code actually takes longer to execute.  If you don’t take the time to think about what is actually going on, then you might be fooled.

Consider a pair of signals, each around 12000 samples. Regulations state that I am allowed to drop (delete) certain samples from those signals before performing statistical operations on them.  The number of points to be dropped might be 2-10%, or up  to 1200 of the points. I have the indexes to be dropped in a third array. For graphing purposes, I need to keep the dropped points in separate arrays.

Now every programmer worth his salt has fallen into the trap of deleting elements 3, 5, and 8 from an array: If you try the straightforward way, you find out that after you delete element 3, that element 5 is not in the same place it was before!  So  you either have to delete element 8 BEFORE you delete element 5 and then 3, or you have to delete element (3-0), then element (5-1), and then element (8-2).

Having fallen into that pothole my share of times many years ago, I avoided it this time by doing the reversing trick: My list of points to drop was known to be in ascending order, so I reversed it, and then did the deletions.  Because I needed the deleted points in proper order, I had to reverse those after the deletion.  Here’s the code:

That worked fine for some time, but while revisiting this code, it occurred to me that it might be faster to manipulate the index while deleting, and avoid the reversals and speed things up.  Here’s the code:

That’s certainly simpler, right?  As one should always do, I applied a Timing Measurement to it. And I was surprised.  I created two arrays of 12000 numbers and an array of 1200 random (0..11999) indexes.  It was consistently 5-6% MORE TIME this simpler way.

But if you stop and think about what’s going on, the reason is clear.  Suppose your signal array contains [0, 1, 2, 3, 4, 5] and you want to delete elements [1, 3, 4 ]

Using method A you reverse the list to get [4, 3, 1 ].
You delete element 4.  {that moves element 5 down – 1 move}
You delete element 3.  {that moves element 5 down – 1 more move}
You delete element 1. {that moves elements 2, 5 down – 2 more moves}
That’s 4 moves that were made in the shuffling process.

Now consider the “simpler” method:
You delete element [1-0].  {that moves elements 2,3,4,5 down = 4 moves }
You delete element [3-1].  {that moves elements 4,5 down = 2 moves}
You delete element [4-2].  {that moves element 5 down = 1 move }
That’s a total of SEVEN moves that were made.

So even though we eliminated three REVERSAL operations, we actually take LONGER because we are doing more work.  The increased amount of data-shuffling was enough to overcome the benefit of removing the reversals.

This was done using a random list of indexes to drop; I’d bet that there are possible scenarios where this wouldn’t hold true (for example if the points to drop were few, and confined to the end of the signal), but given that neither of those will be true in my case, I’m sticking with the original plan – on average it will be faster.

But don’t assume that fewer operations on the diagram means less work !

En masse is a French term meaning “as a whole” or “all together”; treating a group of something as a single unit.   LabVIEW has the ability to treat arrays this way, which can greatly reduce your workload. If you come to LabVIEW from a text-based language, it’s easy to miss the capabilities that are right at your fingertips.

For example, if you need to scale a series of readings into percent of the total (a procedure called normalizing),  then you tend to think:

I need to find the total:

• I need to start with a zero sum     sum = 0.0;
• I need to loop over every element  for (int i = 0; i < nElements; i++)
• I need to add this element to the sum   sum += array[i]

Now I need to divide each entry by the total, to get the fraction of the total:

• for (int i = 0; i < nElements; i++)
• array[i] /= sum;

Now I need to multiply by 100 to get percentages:

• for (int i = 0; i < nElements; i++)
• array[i] *= 100.0;

That’s all well and good, and you could translate that literally into LabVIEW and it will get you the answer you want to see.  But that’s not the LabVIEW way of thinking.

What newcomers often fail to realize is that most primitive numeric functions (the ones with yellowish icons) will accept an array of numbers directly. This goes for basic arithmetic (add,subtract, multiply, divide), comparisons (greater than, less than, MAX/MIN), and many other operations.  It will happily multiply an array of numbers by a single scaler number, to produce an array of numbers.

This has great power to reduce the work that you do as the programmer. Consider the literal translation of the above code:

If that’s as good as it gets then why should I go with LabVIEW?

Well, it does get better.  There is a function in the numeric palette called ADD ARRAY ELEMENTS.  If we replace the entire first loop with this function, then we get to this:

Now for the en masse parts: You can replace the entire second loop with a single operation as well:

Any guesses what we can do with the third loop?    Yes, that’s right:

Now you have SO much more room to add comments about what you’re doing!

Now THIS is what makes you more productive in LabVIEW than in C; your chances for error are far less when you let the en masse operators handle the details, and you don’t even have to think about the details.

But be aware of what’s going on, however; there is no magic here.  Under the hood there is still a loop somewhere.  It’s now hidden somewhat; it’s not as obvious, but the work is still being done.  Don’t let the simplicity obscure the real processing that’s going on.

Here is an example of the normalizing function in use, from the real LabVIEW example examples\general\graphs\charts.llb\Draw Stacked Graph.vi (in LabVIEW 8.6. anyway).

This amounts to the same as our last part above.  In the example, the array given contains five elements and this is executed only once, so efficiency is not a concern.

But consider if the array was 10,000 elements. Don’t forget that the first operation is doing 10,000 divide operations, and the second is doing 10,000 multiplications.  Can you improve things?

Well, certainly! What you have to realize is that, by the associative property of numbers, (X / sum) * 100 is equal to (100 / sum) * X.  You also have to realize that 100 / sum, in this context, is a constant, and therefore needs to be calculated only once.  In effect, you are dividing by sum and multiplying by 100, but you are doing it 10,000 times!

With any luck at all, you get the same answer every time, so you only need to do it once:

THIS is why we use LabVIEW!

NOTE:  En masse is my term for this feature, it is not an official LabVIEW term.

Usually, in a data acquisition program,  all the signals you measure are “live”, meaning they represent the current conditions at the time they are sampled. However, in some cases you might have some signals which are not live, but delayed. For example, suppose you’re measuring engine operation, and you have gas analyzers sampling the exhaust airstream. These analyzers conduct the gas from the measurement point in the airflow system around the engine to the analyzer mechanism itself. This gas flows through the sampling tubing at a specific rate, and therefore arrives at the sample point at a specific time AFTER it left the main airstream. These delays might be multiple seconds in duration.  There could be other reasons for this delay, such as an echo-measuring device, or the mechanical response time of some piece of hardware.

For analysis purposes, you want to look at the engine holistically and see causes and effects. If the speed changed at a particular point in time, you want to see the CO concentration change as a consequence. But unless you do something to compensate, the change in gas concentration will lag far behind the change in speed that caused it in graphs and tables, making it difficult to judge consequences.

To arrange the signals back into time-synchronous alignment, the solution is to think of ALL signals has having TWO delay lines: one physical, and one logical.  In our example case, the physical part is the gas piping conducting the gas. The logical part is completely within the  DAS software. If EVERY channel has a physical delay time (even if it’s zero), and if EVERY channel also has a logical delay time (even if it’s zero), and if EVERY channel has the sum of the physical delay time and the logical delay time equal to a constant, then the signals coming out of the logical delay lines will be time-aligned.

Example

For example, consider three channels: Channel A records data “live”, i.e. no delay. Channel B has a delay of 3 seconds. Channel C has a delay of 5 seconds.

We want to make the entire chain (physical + logical) the same length for all channels, so we pick the longest delay (5 sec) and make that our total delay time. For channel A, which has no physical delay, we have to have a logical delay of 5 sec. For channel B, which has a physical delay of 3 sec, we add a logical delay of 2 sec for a total of 5. For channel C, which has a physical delay of 5 seconds, we have a logical delay of zero. 5+0 = 3+2 = 0+5, so every channel has the same delay, when we count both physical and logical.

The logical delay is implemented in queues with every channel having its own queue. Every sample received goes into a queue belonging to that channel; and we pull out one sample from the end of the queue to be recorded. Since one goes in and one comes out, the length of the queue never changes. The initial length of the queue corresponds to the delay time we need for a particular channel. If a queue is empty (of zero length), then the sample we put in and the sample we take out are the same sample. If the queue is of length 10, then the sample we get out was taken 10 sample times ago.

At configuration time, the queues are set up, and populated with zero values to set their length according to how much delay is required. After that, the sample process means putting a sample into a queue and taking one out for use.  Every channel operates the same when actually sampling, it’s only the configuration where they differ.

Note that if you are recording a TIMESTAMP value, then the TIMESTAMP signal requires its own queue as well, so that it comes out in alignment with the signals. Of course this means that the data does not become usable until all the zeroes have been flushed out of the queues. That will happen when the longest delay time has expired. We can judge this by putting a RECORD signal into its own queue. We don’t actually record data coming out of the queues until the RECORD signal coming out is true.

You can keep the data flowing into (and out of) these queues all the time.  When you want to start recording, set the RECORD flag TRUE. When you want to stop recording, set the RECORD flag FALSE.  Pay attention to the value out of the RECORD flag’s queue, and act appropriately. (Don’t act on the RECORD flag itself, act on the delayed RECORD flag).

This also means that when the test is over, there is still data remaining in the queues. We have to keep recording until the valuable data has worked it’s way through the queues, meaning 5 seconds after the last sample (in the example case).

In LabVIEW, there are three kinds of files:

• TEXT files. Ordinary text, stored in human-readable form, with spaces and line feeds, etc.
• BINARY files. Raw information stored as machine-readable information. A 32-bit integer is stored in 4 bytes. A double-precision number is stored in 8 bytes.
• DATALOG files. Structured data, suitable for quick transfer between a memory structure and the disk file.

I have long used Datalog files for configuration data. They offer several advantages:

• EXTREMELY simple reading/writing. No matter how complicated the data structure, you just open the file, write the structure to it, and close the file. No muss, no fuss. I have a project where the datalog file is a single record of maybe 200 k Bytes. It’s still all done with a single WRITE FILE call.
• Being binary, users are reluctant to open it in their text editors and twiddle about. You won’t get a service call to figure out that the user set the serial port to COM1.76 and the alarm level to 0 ( But it was working perfectly yesterday! ).
• Compactness. A DBL is 8 bytes. The value -12.3456 is 8 bytes, not even counting the delimiters separating it from it’s neighbors. If space is truly an issue, use SGL (4 bytes) instead of DBL.
• If your panel is set to allow a number to vary between 10 and 20, then that’s what gets stored. You don’t have to check every number in the file to see if it’s in range. Since you wrote it there, it’s correct.
• Format checking. Even if you use the standard file dialogs for choosing files (rather than a custom one), you can set it so that it will show only files of the correct type. The user has fewer wrong files to choose from, therefore the odds of a mistake are lessened.

There are a couple of disadvantages, though:

• Rigid formatting. The thing that makes it so easy to read and write, turns around and complicates things when it’s time to revise the format. If you add or remove so much as a single item, or re-arrange the order of things, then the old files are not readable anymore. You can attempt to compensate for this by adding spare fields at the start, but if you make a change to the format, you will have to make an updater which reads the old files and writes new ones in their place, or else all the old files are worthless.
• Non-portability. The tamper-resistance feature can be a disadvantage if the file must be available in other (non-LabVIEW) applications. For this reason, datalog format is best suited to files that have limited, well-defined uses.

Typically, a data-acquisition program, when used over a reasonable period of time, needs a configuration file to define which channels to use, what their scale factors are, their names, and units, etc., etc. The data recorded in “Run 107″ was recorded using “JOEs setup”, but the data recorded in “Run 108″ was recorded with “JOEs Other Setup”. So how do you keep the files paired? You don’t. You can try various naming schemes, but sooner or later, some mistake will leave the user with a missing or mismatched CONFIG and DATA file pair.

I avoid that whole scenario by including the config data structure inside the data file. Every data file contains the config used to record it. You just put the config cluster inside a large cluster that includes your data, and record that. There is no question about which scale factor was used on the flatistrat channel, because it’s recorded right there. It’s a bit wasteful in terms of disk space, but not terribly so.

Datalog + Binary = Hybrid

So, given all that, suppose you have a LOT of data to record. The config data is a small portion, and the data is huge. There’s a problem with the idea of a data file being a cluster containing the config structure and the data. And that problem is memory size. To write a cluster to a datalog file, you have to have the data all in one place. If your data is stored in some other place as it’s acquired, then writing the file means making a COPY of your huge data and putting into the file cluster before writing the file. That’s wasteful. And back in the days when 8 Megabytes was all the RAM a machine could hold, and my clients needed to record more, it wasn’t even POSSIBLE.

To solve those issues, I invented a hybrid file. That term is my own label; it is not an official LabVIEW term. The idea is that you write the config data as a DATALOG file, and close it. Then you open the SAME file as a binary file , skip past the datalog portion, and write binary data. You get the benefit of both worlds: it’s easy to read / write the config header with ordinary datalog operations, and it’s easy to read the binary data with binary operations. You can write the data as you need to; you don’t need to make copies. You can write more data than you have RAM for. You just have to remember that the file doesn’t start at offset zero. It’s perfect, right?

Almost.

You have to figure out where the start of binary data should be, and that’s not trivial. LabVIEW’s DATALOG files include their own header, and the structure and size of that header is not public information. However, you can make some deductions. Since the FILE DIALOG can discriminate between datalog files of different structure, the structure format has to be embedded in the file itself. So what I do is flatten the datalog portion to a string and get it’s size. I take the TYPE STRING (which is not really a string) that comes out of the FLATTEN function, and flatten that and get another size. I add those two sizes together, and round it UP to the next highest multiple of 4096 or so. That works for any size structure, as it includes an estimate for the Datalog header, as well as our own header.

When you read the file, you do the same thing, and compute the offset where the binary data starts.

One more gotcha. Occasionally, National Instruments changes the format used in Datalog files. Usually it’s a minor change, and usually LabVIEW handles it automatically. You may have seen the message “This file was recorded using an older version of LabVIEW, it must be updated to be read. Do you want to update it?”. All that is well and good if it’s a plain Datalog file, but if it’s a hybrid file, LabVIEW doesn’t know anything about the binary data you’ve stuck on the end. So it will open the datalog portion, and re-write a new datalog portion, truncating everything after that, including your data. Beware.

Still this method brings more benefits to the table that it brings problems, so consider it for your own projects.

Enjoy.