2017-07-13

Building Docker "Toolbox" for 32bit host system: docker-cli 17.07 and docker-compose 1.15

Background story is simple: I've got x86 Windows7, and wanted to run docker. Docker-for-Windows says it needs Windows10Pro or better due to its improved Hyper-V support. Eh. Fortunatelly, they provide so-called "Docker Toolbox" for other systems that does not match requirements for normal installation. Docker Toolbox is essentially a VirtualBox setup that will host a VM inside which Docker will be able to set itself up. Alright! I tried. Guess what! Docker Toolbox needs 64bits.

I couldn't understand why it's not available on x86, it sets up a goddamn VM, and VirtualBox works on 32bit as well! I searched a bit and I found out that there were some recent patches and updates to enable running boot2docker image on 32bit. I think Stefan Scherer developed a patch allowing that. It was ~1 year ago, so I searched for some instructions how to apply that. I think my start point in that matter was this article and some further articles indicated there. Their story was simple: everything works, you just need to prepare manually the initial VM, you get rest from chocolatey repository.

Unfortunatelly, these articles are a bit outdated and it turns out that today there's much more work to do than was described. Installing docker-machine via choco fortunatelly indeed worked fine! However, when I wanted to install current stable Docker version 17.06 on my x86 machine - Guess what! - since version 17.06 they stopped providing x86 builds, and more, they seem to have removed old x86 builds as well (only x86_64 in https://download.docker.com/win/static/ subfolders). Hi Murphy!

But, since you are here, you're probably interested in how to get them back. Skip to the end of this text for a link to binaries, but be warned that at the time of reading this, they may be old, and I probably won't be rebuilding and updating any time soon.

Setup tools and freebies

Off we go then:

Install VirtualBox (https://www.virtualbox.org/wiki/Downloads)
Install chocolatey (https://chocolatey.org/install)
Install docker-machine via choco (choco install docker-machine)
Create a default VM through docker-machine (docker-machine create --driver virtualbox default)
Check if the VM works (docker-machine status default)
Check if you can get the config (docker-machine env default)
Store the config for later (docker-machine env default > env.bat)

If docker-machine does not work for you, panic. Maybe you will need to build it yourself as I did with docker-cli and docker-compose. I didn't have to, so I don't have a solution/instruction for that, sorry.

Rest of the article assumes that docker-machine still supports x86 hosts and that installing docker-machine succeeded.

Try installing docker (choco install docker)
Try installing docker-compose (choco install docker-compose)
Check if it works (docker version)
Check if it works (docker-compose version)

Most probably, last two points will fail with "program is not valid win32 application" error or similar. This means that choco has found only x86_64 version and installed that for you, and, well, it's unusable. Uninstall them by

choco uninstall docker
choco uninstall docker-compose

Btw: If you don't need the latest version of docker, you can try installing some older one. I found out that 17.05 runs fine under Windows7 32bit - you can get it by choco install docker --version 17.05. However, at the time of writing, you won't get docker-compose that way, since they only provide x64 binaries. If you found yourself an acceptable and working version of docker, skip the next chapter and fast-forward to buiding docker-compose.

Building docker-cli for x86

Let's get the basic docker commandline utilities, the "docker-cli" project. For that, we have a small chicken-and-egg problem, since the build process actually required a working docker installation. However, we already do have a docker-machine up and running, and that's exactly what's neeed.

But first, we need to get inside it. If you ran VirtualBox GUI, you will be able to simply show the VM's screen and use it directly as a terminal. However, I don't recommend it. Clipboard will not work, for example. I prefer to use SSH, any client will work, I like PuTTY. Peek the env.bat from point (7) and check the IP address of the VM and connect to it through SSH. At the time of writing, the default login:pass is docker:tcuser. Watch out - while default VM's screen from VirtualBox, which logs you in as root - the SSH will log you in as docker. Some commands may require elevation, but you can even sudo sh and be root..

Once you have access to the terminal, check if it really came with some default tools and also check if the network access works.

docker version
git --version
ping www.google.com

If anything of those does not work, you will have to fix that somehow to continue.
Assuming everything's good, fetch the sources:

cd /root
git clone https://github.com/docker/cli.git
cd /root/cli

and do not build it yet, you would just waste your time.
At the moment of writing, the default scripts will build you linux, osx, or windows version, but the windows part is configured to x64, so it needs a few updates.

At some point of time I found this articles, which gave me a great kick-off in that matter:

this article "Docker Daemon for x32 architectures"
or that article "Installing the Docker Client CLI on 32-bit Windows" (it's from 1st April, but no fools there)
https://gist.github.com/prateekgogia/05f058bafbccc2478fcc/2f6c7b632a7b391618a3ad6cfdce949142efa3c2

I think I got that last link it from first article, and then followed GIST updates. Anyways, it was supposed to replace the default 'Dockerfile', however.. if I remember well, the build scripts have changed since it was last posted, and I'm pretty sure that eventually I didn't use it at all. I don't remember, and I can't retry this right now. I mention it there only because these articles were important and because it may be needed for someone in case I screwed up this instruction.

When I noticed that I cannot use that Dockerfile from prateekgogia's GIST, I looked around and noticed that docker-cli is written in GO-LANG. I thought that this one should have not a single problems cross-compiling for win32 so I just changed the build scripts to target different platform:

cd scripts/build
edit the file called windows

At this moment you will probably notice that there's no software.. unless you want to cat/sed, let's install some. It's not debian/ubuntu/redhat/etc, we're on TinyCoreLinux, package management fortunatelly exists, but repo is limited. I found two editors capable of working in terminal:

tce-load -wi nano
tce-load -wi vim

One is enough, pick what you like the most or dislike the least.
Once you have editor, edit file called "windows" and change BOTH:

CC= from x86_64-w64-mingw32-gcc to i686-w64-mingw32
GOARCH= from amd64 to 386

Remember to save that file.
In case you noticed 'w64' in the mingw package name, it's OK. That version can build both x64 and x86 binaries.
Now, try building the cross-compiled profile:

cd ../.. (so you're back in 'cli' folder)
make -f docker.Makefile cross

The line (23) is taken from readme available at docker-cli's github page. If something goes very wrong, maybe it has changed since I wrote this text. Just go there and check what's the current startup line, maybe you will even find better instruction than mine here.
Anyways, this line will, again, most probably, fail due to make command not being present. Just install it in a similar way to installing text editor:

tce-load -wi make

and retry line (23).
It will take some time. Build process will download mingw, download and run some containers for the actual build process, and so on. After it built everything, it may end up with crash on copying the final output .EXE file. I might have mixed the facts with how building docker-compile worked, so I'm not sure if it happens at all, but if it does, just check the 'dist' folder for .EXE file. If it has some x64 or amd64 in the filename, don't believe that file name - we changed target architecture in build scripts, and that file name may simply be wrong.

Copy that file to your win32 host machine to some folder of your choice
Rename to docker.exe
Try running i.e. typical docker version to check if it works.
Add location of that file to PATH environment variable

Since you have now both docker-machine and docker commands available, you may use now them to clean up the containers and images that were used by the build process. That's optional, you can leave them there if you plan to do more builds in future.

I suppose you also may ignore the root docker container that you used as the entry-point terminal to download the sources and edit configuration files. That container seems to be automatically purged and cleaned at every (re)start of the docker-machine.

Btw. If docker complains about missing settings, use env.bat file to set them. It may be a good idea to refresh that file at times, see line (7).

Building docker-cli for x86

It turns out that docker-compose is written in Python not in GO, and that at the time of writing this text, Python tools that build .EXE files lack support for cross-compiling between x64 and x86 (or I may have just failed to find how to do it - however many articles claim that this option in PyInstaller was recently removed). If you want to get x32 executable, you have to run the build process on x86 machine. I suppose we already have one: your host machine that you want to run docker on.

git clone https://github.com/docker/compose.git
view script/build/windows.ps1 and follow the procedure stated in the opening comment
Install Python 2.7.x if you don't have it yet (https://www.python.org/downloads/)
Ensure that python and pip are both at PATH (if not, add something like "C:\Python27;C:\Python27\Scripts" to the PATH)
pip install virtualenv
powershell Set-ExecutionPolicy -Scope CurrentUser RemoteSigned
cd compose (the root folder of the sources)
powershell -file .\script\build\windows.ps1

Again, it may take a while to build. It's all in Python and it uses PyInstaller to generate .EXE file, and it is smart enough to notice that your current OS is 32bit and will build you a 32bit executable. Again, it may fail at the very end, during file-copying. If it does, look for a new .EXE file that should show up in the dist folder. Probably, it's name is docker-compose-Windows-x86.exe and that's the problem, because build process is hardcoded to copy docker-compose-Windows-x86_64.exe (last lines of the .ps1 file). Anyways, we don't care. We got out .EXE file.

Copy that file to some folder of your choice
Rename to docker-compose.exe
Try running i.e. typical docker-compose version to check if it works.
Add location of that file to PATH environment variable

Now, optionally, you can delete the folder with docker-compose sources, and uninstall Python27, and remove Python and Python\Scripts from PATH.

Disclaimer

As usual, there may be some minor gaps and noise in the process I described above, I wrote this instruction basing on my notes gathered through the last few days. I made many consecutive attempts before achieving the goal of having x86 versions of docker-machine, docker-cli and docker-compose, so I might have forgotten/overlooked/mixed up some facts, like forgetting about tce-load -wi something extra. However, I'm sure that overall process looked like I described above, and I got it working. I managed to run hello-world container, ubuntu container, and ubuntu32 container, and manage them, so.. seems to work.

TL;DR

I built them, so ~~you can get them here~~:

Please be aware that those builds were done directly from the main git repos, so they are DEV/EDGE builds, not STABLE builds. Since the version numbers are not frozen (and i.e. docker-cli got already some new commits for 17.07 so if yo do your build of 17.07 you'll get a newer 17.07 than my 17.07), I included a timestamp and GIT SHA at the end of the filenames so we can later identify what's the exact source code version. Both tools also seem to identify themselves via version command, including commit hash.

Update: ~5yrs passed, I removed those builds from file hosting. They were probably criminally outdated already anyways. If you relied on them in any way, sorry, deleted permanently.

2016-10-17

Getting faster transfer speeds when reading measurement results from Rigol DS1054Z

After reading countless reviews, issues discussions, comparisons, price tags and all, I finally bought a Rigol DS1054Z oscilloscope. It was some time ago, somewhere in June/July this year. Until then, I used a Hantek DSO-2090 "PC oscilloscope", or rather, a kind of a data logger shield connected by USB, which means you can't actually do a thing unless you have it connected all the time to a computer with a proper software running. No display, no knobs. Just a box with three BNCs for CH1/CH2/EXT. It has its issues, but I learned a ton. Since it came with its own proprietary software (not that bad actually) and some API with almost no docs, it was obvious that device can be fully programmatically controlled from the PC. I researched the API on my own and started writing my own software for it. But that's another history, maybe I will have time to write it all down some day.

Anyways, I got really used to it over time and I wanted to check if I can do the same with a "real oscilloscope". That was one of the reasons for selecting DS1054Z. Its price is relatively low and it has a quite well documented programmatic access. When I started to play with DS1054Z, the first thing which surprised me was .. I couldn't reliably download the measurements results from it.

Note: I use Windows 10 and NI-VISA/IVI drivers. I know it's a bloat for my use case. I'm not using LabView, I'm not building automated lab (maybe later). I have this one oscilloscope, a PSU (axiomet, that one I wrote earlier about), and that's all. I took these drivers because .. they seemed to be default ones suggested by manufacturer. I would really like to strip that 1GB software package down to the actual few megabytes of device drivers needed. I even lost a few hours trying to rip them out, but failed, and I don't care about the harddisk space much enough to spend more time trying. I also didn't want to start researching raw device protocols like I had to with Hantek.

IMPORTANT: I actually have not verified data that was read from the device during these tests; in some or all cases I could have got a total garbage, or a mixture of all channels, instead of the channel I wanted to read from. The "fun fact" from part two indicates such possibility. I was focused on communications and simply didn't have time to verify the data yet. When I read from 1-chan or 2-chan modes and then looked at raw binary data, it looked fine. But that doesn't tell anything. I need to generate some images and compare to on-screen data to be sure. Verifying the structure of received data and determining which modes are usable and in what way is the next thing I'm going to investigate. I will remove this warning text afterwards.

Intro: basic setup

My current DS1054Z firmware, 00.04.03.01.05 "SP1", built 2015-05-26 08:38:06, has a really interesting way of determining how much data can you fetch from it in one go.

The device has a memory of ~24M samples. It can be used for a single 1-channel measurement. I started with that and tried to download all 24M samples. According to the "MSO1000Z/DS1000Z Series Programming Guide 2014", it is not possible right away.

First of all, as noted in the Guide for :WAVeform:DATA? command, there are various "areas" that the data can be downloaded from. Namely, at least two: screen and memory. By default, device will return samples from "screen" (:WAVeform:MODE NORMal). You can get them always, even if the device is working, but the data returned is .. the data you currently see on the screen. It's fast, it's easy to fetch, but I wanted to get whole 24M of the measured data, not just a ~1000pts (1K) post-processed fragment of it.

To access the other buffer called "waveform data from internal memory", you have to switch the :WAVeform:MODE RAW option first. I'll skip the MAX option since it doesn't change much. Also, the device cannot be measuring at the same time, it has to be in a "STOP" state. Basically, if the device was left in 'auto' or 'waiting for trigger' states, you can assume that the data of the previous measurement has already been (partially or fully) overwritten by new samples, and you can safely get only the screen data.

Another intersting thing is that the :WAVeform:DATA? won't usually return you the whole data. It actually returns only a part of the data taken from a range set up by a pair of :WAVeform:STARt xxx and :WAVeform:STOP commands. For example, here's an attempt to read two consecutive chunks of 1000 (1K) samples:

:WAVeform:STARt 1
:WAVeform:STOP 1000
:WAVeform:DATA?
> #9000001000........
:WAV:STAR 1001
:WAV:STOP 2000
:WAV:DATA?
> #9000001000........
...

As you see, the 'addresses' start at 1 (not zero), and the START-STOP range is inclusive-inclusive. As you may guess, the last address is either 12000000 (12K) or 24000000 (24M) (depending on your device feature state).

There is a reason I started with this example. You cannot just set START=1 STOP=24000000 and have fun:

:WAV:STAR 1
:WAV:STOP 12000000
:WAV:DATA?
> #9000000000\n

Here, I tried to read too much in one go. Device didn't raise an error. It simply responded: ok, zero bytes for me - and it actually included no measurement data in the response.

There is a limit to the amount of data that can be read in one block. The device communicates via USB/etc and it seems to have a limited communication buffer which simply cannot be exceeded. If you want more than that, you need to read it block-by-block, like in the first example.

Also, it's good to note the :WAVeform:FORMat option. As I saw in various articles, many folks out there use ASCii option so they can easily process the data in scripts, spreadsheets, etc. But this means that the device has to print out the text instead of raw data, and the text eats the buffer much faster. I wanted to download as much data as possible, so I use :WAVeform:FORMat BYTE. The measurements are 8bit anyways and this format saves much comms time.

I also want to use as large blocks as possible. For example, I could read using blocks of 1000 (1K) samples as in that first example, but then to download whole 24M memory, it would need .. 24000 "start-stop-data?" queries, but it would mean a ridiculously high (really) waiting time for me. Something like .. 2 hours .. probably. That's not what I would like to wait after each measurement :) Fortunately I don't need to, since that 1k window was just written there for an example.

Sadly, "MSO1000Z/DS1000Z Series Programming Guide 2014" (that's the one I used at first) doesn't give any hints on what are the buffer limits. I later found "MSO1000Z/DS1000Z Series Programming Guide 2015" and there you can find a table (page 2-219):

BYTE  -> 250000
WORD  -> 125000
ASCii ->  15625

Even from this table you can see that picking ASCii mode means lowering the transfer rate to 15.6 kilo-samples per query. Comparing to RAW/BYTE mode and its 250K, this means over 16x slower transfer. For those that don't "fell" the multipliers: instead of a 1 minute, you would need to wait 16 minutes.

As I said, I found 2015 guide much later, and I already had discovered this myself, as well as the fact that this numbers are not accurate. These numbers are safe, meaning, you can (probably) use this numbers at any time.

For a single channel, I am able to successfuly use ~1180Ks blocks. That's over 4x the number mentioned in the docs. I have not hacked the firmware or hardware. It's just that the docs didn't want to delve into really detailed details of how to get that speed.

Comparing things to the value suggested in docs:

    Channels: CH1
        Mode: RAW
      Format: BYTE

      Memory:   24.0M   12.0M    7.5M*   6.0M*   2.4M*   1.2M
Block=250K
    Queries*:     95      47      29      23      10       5
        Time:   56.6s   27.2s   17.1s   13.2s    5.7s    2.7s
Block=1180K
    Queries*:     21      11      7       6       3        2**
        Time:   16.5s    8.5s    5.3s    4.3s    1.8s    0.9s**

*) these acquisition memory depths are not available "by the knobs" on the device front panel, hovewer you can get them using "AUTO" memory depth
**) sadly, the best transfer I got was 1180K and here the source memory a tiny bit larger than that, so two queries were needed

I'm not including results for memory size smaller than 1.2M simply because it would all fit into one query. I'm not including results for smaller block sizes, because the programming guide gave such a suggested value, I see no point in limiting the block size to lower than that. Also, I'm not including comparisons to ASCII format, since.. I'm not a masochist. I want the data, I don't want to sit and grew old waiting for it.

My test code isn't super-optimized so you may be able to squeeze a better transfer times. You could probably remove some assertions and some noncritical commands sent to the device during a single "query", but the point is that the higher block size you can get, the lower your waiting time will be. It may not look like much for 1.2M depth, but for the higher ones it really makes a difference.

However, to get 1180K blocks in a reliable way, you need to prepare both yourself and the device for that first.
No hardware hacks needed.

What's the problem?

My current DS1054Z firmware, 00.04.03.01.05 "SP1", built 2015-05-26 08:38:06, has a really interesting way of determining how much data can you fetch from it in one go. It consists of two really inobvious parts.

First of all, it's not a nice value of 1180k, but actually, it's a value between 1179584..1179647 samples per block. Not a random value. You can see that the 'range' is exactly 64 bytes long and that lower bound is divisible by 64b, and upper bound+1 is divisible by 64b as well. In hex that's 0x0011FFC0 and 0x0011FFFF. I suppose that it comes from device's internal memory architecture, grouped in segments of 64 bytes. I have really no idea though, just guessing.

Now, that latter value, the upper bound, looks very tempting! (Prett-y shin-y round-y 0x120000 tempts too, but it is not available at all). However, using 1179647 requires some very precise conditions to be met, that even I, after researching it and finding out the rule, I decided to drop it and stick to the lower bound value, 1179584/0x0011FFC0 that is almost always available. (that 'almost' word is the second big part of the mystery, it's covered in the next section of this article)

When I first investigated it, I didn't know the limits, nor the window size. From a few manual attempts I knew how a successful read looks like and also I knew that if the requested block size is too large, the device will respond in #9000000000 empty response instead of some helpful kind of a range-error. I wrote a small application to scan various setups and bisect the blocksize. It would start at some position, with blocksize=1 and blocksize=24M and would try intermediate values until blocksize=N is ok, and blocksize=N+1 is not ok.

Here's an example of results:

measurement 01, reading at START=1:       found best STOP=1179587      => max read block size=1179587
measurement 01, reading at START=1000:    found best STOP=1180611      => max read block size=1179612
measurement 01, reading at START=1000000: found best STOP=2179587      => max read block size=1179588

Surprise! I would expect that transfer capabilities would be the same, regardless of position. I left the scanner overnight and let it scan positions at random. It found out that the quite probale min/max blocksize values were 1179584 and 1179647. (Initially I was sure that the max is 1179648, but that was due to off-by-one error due to the :WAVeform:STARt beginning at 1)

I also made some finer-grained sequential scans and they has shown that numbers in range are not random at all and that they form a saw-tooth shape:

...
reading at pos=1000:    bsmax=1179628        |
reading at pos=1001:    bsmax=1179627        |  you can see the available blocksize falling by one
reading at pos=1002:    bsmax=1179626        |  for each increase in position; all on 1-by-1 basis
reading at pos=1003:    bsmax=1179625        |
...                                          |  but then, suddenly the value jumps
reading at pos=1042:    bsmax=1179586  (-61) |  from 1179584 to 1179647, and then falls again on 1-by-1 basis
reading at pos=1043:    bsmax=1179585  (-62) |  then the cycle repeats
reading at pos=1044:    bsmax=1179584  (-63) /
reading at pos=1045:    bsmax=1179647  (-00) \
reading at pos=1046:    bsmax=1179646  (-01) |  the length of such 'monothonic window' is 64 samples(bytes)
reading at pos=1047:    bsmax=1179645  (-02) |  starts with highest possible blocklength=1179647
...                                          |  ends with lowest possible blocklength=1179584
reading at pos=1107:    bsmax=1179585  (-62) |  
reading at pos=1108:    bsmax=1179584  (-63) /
reading at pos=1109:    bsmax=1179647  (-00) \
reading at pos=1110:    bsmax=1179646  (-01) |  ..window of next 64 samples/bytes, and so on
...

I thought, wow, so in fact the memory is segmented and it's just that a single query just can never cross a segment boundary. Or something like that. Or more likely something different than that, since this description doesn't fit at all, as I was reading over a one million samples in one go and the segments ('windows') seem to be 64 bytes long, so each 1M query crosses a ton of them. Well, nevermind ;)

Anyways.. sawtooth pattern holds across the whole 24M memory.
But that's not all to it.

I played a little with the DS1054Z, made a few measurements and tried to fetch results using the same windows and offsets, and I couldn't get it working. Just to be sure I noted the values correctly, I ran the scanner again and here's what I saw:

measurement 02, reading at pos=1:       bsmax=1179603
measurement 02, reading at pos=2:       bsmax=1179602
...
measurement 02, reading at pos=999:     bsmax=1179629
measurement 02, reading at pos=1000:    bsmax=1179628
measurement 02, reading at pos=1001:    bsmax=1179627
...
measurement 02, reading at pos=999999:  bsmax=1179605
measurement 02, reading at pos=1000000: bsmax=1179604
measurement 02, reading at pos=1000001: bsmax=1179603

Please compare it with the first measurement and positions 1/1000/1000000 above. Suprise#2! Although the values are still in the same bounds, they are different. Fortunatelly, the sawtooth pattern is still preserved. It "just" "starts" in different place.

Actually, it turns out that the sawtooth pattern gets offsetted after each single triggering and capturing another waveform.

I think that, maybe, while waiting for a trigger condition, the device constantly samples and writes to the sample memory in a ring buffer scheme, and when trigger condition is detected, the device simply runs forward and stops to not overwrite the trigger position (minus configured pre-trigger amount to be left visible). That's a pure guess of course. That's how my old Hantek DSO-2090 worked, but then it didn't have any 64-byte windows. It just returned its tiny 10k or 64k of samples.

Whatever is inside the device, the facts are that after each measurement, a random number between 0-63 is selected as the 'monothonic window offset', the 64 byte window at :WAVeform:STARt 1 is shortened by this value, and then a normal sawtooth pattern follows with full 64 byte windows till the end of the memory (where upon at the very end obviously the very last window will also be shorter).

Knowing that all, we can sketch a following readout procedure:

we know the min/max blocksize (constants)
we can learn the current offset by simply trying to read at START=1 and checking which blocksize will be accepted; we can check it by bisecting in about 6 attempts (may be time consuming if we often hit good sizes and get 1M response), or we can start with max blocksize and try-fail-retry decreasing blocksize by 1 each time (max: 64 fast "zero-length" responses)
knowing the window offset, we can easily determine all the sawtooth peak points
keep the first partial block that was read at the time of determining the offset
read all blocks from second to next-to-last at max blocksize=1179647
calculate how much data is left, and read the last block

Assuming a typical case when the random offset is not zero (assuming flat distribution, that's 98% of cases), we get floor(MEM/maxblocksize) full reads followed by +2 partial reads [read=set start, set stop, query data]. Plus some queries to learn the current offset value. So, for full 24M that's X+20+2 reads. Nice!

However, as you might notice, the table from the Intro section claimed that my code was able to read 24M in 21 queries. How come?

That's simple: I was lazy and I didn't implement it. What I did was take the lower bound of the sawtooth pattern, 1179584. This value allows you to completely ignore sawtooth - just because the lower bound value is valid throughout the whole memory - and read the data as it goes in blocks of 1179548 bytes right from the START=1 till the end, where partial block read will occur. That means floor(MEM/minblocksize) followed by +1 partial read, which for 24M gives .. 20+1. And we drop the 'X' since we don't care about the offset. Well, children please don't listen now, sometimes laziness pays off!

Availability of 1179K blocksize buffer

If you pick now the magic 1179584 number and try it out yourself, you have a high change of failure. That's because it's the "max value". It does not mean that your device will handle that right away. We still have the second part of the mystery to see.

Since I used 2014 Programming Guide (the one with no hints) I had to guess the blocksize. I remember I tried 12M at first, failed, then 5M, failed, then 2M, failed, then 1M, failed, then succeeded and got data in a stable way at some blocksize around 500k. Few days later I returned to this topic only to find out that now I can use those 1.0M blocks as well. I was suprised, but hey, I got bigger blocks now, great.

However, I remembered seeing "500k" before, so when after some time my device started rejecting 1180K and started to claim that the highest transferrable blocksize is 580K I felt like "I knew it would return".

Here begins a long story of trials and errors, many overnight scans and trying out different setups, which led me to creating a following table:

------- no channel*---------    Marker [+] shows trigger setup
[+] ::: ::: ::: ::: -> 1179k    <- TRIG=CH1
::: ::: ::: ::: [+] -> 1179k    <- TRIG=AC
------ one channel ---------    If trigger is on active channel, it can be ignored
CH1 ::: ::: ::: ::: -> 1179k    <- TRIG=CH1
::: ::: CH3 ::: ::: -> 1179k    <- TRIG=CH3
[+] CH2 ::: ::: ::: ->  580k    <-- **
[+] ::: CH3 ::: ::: ->  580k    <-- **
[+] ::: ::: CH4 ::: ->  580k
------ two channels --------
CH1 CH2 ::: ::: ::: ->  580k
CH1 ::: CH3 ::: ::: ->  580k
CH1 ::: ::: CH4 ::: ->  580k
[+] CH2 CH3 ::: ::: ->  290k    <-- **
[+] CH2 ::: CH4 ::: ->  ....    not tested yet
[+] ::: CH3 CH4 ::: ->  290k    <-- **
---- three channels --------
CH1 CH2 CH3 ::: ::: ->  290k
CH1 CH2 ::: CH4 ::: ->  ....    not tested yet
CH1 ::: CH3 CH4 ::: ->  ....    not tested yet
[+] CH2 CH3 CH4 ::: ->  ....    not tested yet
----- four channels --------
CH1 CH2 CH3 CH4 ::: ->  290k
----------------------------

Notes to the table:

- effects of 'REF' and 'MATH' were not checked
- effects of various memory and timebase modes were not checked
- (*) actually, when you turn off all channels, most of the things behave as if CH1 were active but just not displayed; I mean, even [SINGLE] button works and actually TRIGGERs and refreshes the waveform
- (**) fun fact: the tool I build for testing had the data-reading queries hardcoded to read from CH1. As you can see in the table, it successfully read the data when CH1 was the active channel was i.e. CH3.. I wonder, what was actually read there? :)

As you can see, most of the times I had trigger on CH1. It had me confused for some time, and the relation between active channels and max-blocksize was unclear, until I remembered about the trigger and included it in the table.

To sum up the table, the rule for determining blocksize is quite simple, count the active channels, including trigger, then if the result is:

1 channel  -> max blocksize = 1179K / 1 = 1179K
2 channels -> max blocksize = 1179K / 2 =  580K
3+channels -> max blocksize = 1179K / 4 =  290K

If you compare Programming Guide from 2014 to 2015 version, you can find that in the latter one, in comments for :TIMebase:DELay:SCALe there's a set of rules to calculate so-called amplification factor. The rules for its channel sum works seem to be exactly the same as seen here, with a small note that TRIG=AC counts as ZERO (IIRC it is not mentioned in channel-sum rules).

But if you think you can just switch some channels on or off to get a higher blocksize, you're wrong!

During the runtime of the device, the blocksize limit seems constant. I tried various things, including disconnecting USB, resetting by *RST command, resetting by [CLEAR] or [AUTO] buttons - nothing seems to change the blocksize limit once it is set.

It seems that the blocksize limit is determined .. at the BOOT TIME

You can only guess how long it took me to figure that out. Sadly, I haven't wrote it down.

As you know, the device by default, remembers last-used settings. I think you can change that behavior somewhere in the Utility and set it to revert to some preset config instead. Anyways, what counts here is, how many channels are active during boot time. If you turned off your DS1054Z having 3 channels active (or 2 channels and trigger on third), then your next session tomorrow will have max.blocksize=290K. Whoo.

An interesting thing is that this works both ways. Once you turn off the device in a zero- or single-channel state, it then boots in 1179K and it seems to keep that mode until shut down, even turning on all four channels during doesn't degrade the max transfer size. No needed to adjust any other options, just turn off all channels before turning off the device.

TL;DR

0) all detailed information contained here applies to firmware, 00.04.03.01.05 "SP1", built 2015-05-26 08:38:06; I have not tried other versions yet

1) use RAW/BYTE mode to save bandwith; BYTE is not very convenient but it is not that hard to calculate actual values from it

2) magic numbers for transfer size limits:
- absolute max data payload size: 1179647 samples(bytes), but please DONT USE IT; explanation is at the end of What's the problem part
- easier to use max data payload size: 1179584, it's almost as high as the absolute max, and with it you can ignore many irritating things

3) if your acquisition memory depth is higher than blocksize, you have to make several batches of 'start-stop-data?' commands

4) when reading, memory adresses start at 1 (ONE). Not zero. Watch out for off-by-ones. START and STOP commands set the data range, both values are inclusive. When reading at START=XXX, you have to set STOP=XXX+blocksize-1. Watch out for off-by-ones again. Really. I lost several hours tracking +/-1 errors.

5) if your device rejects 1180K blocksize, turn off all channels then power off the device. After you turn it on back, it should be good to go at 1180K.

2015-08-31

meteor + sanjo:jasmine + urigo:angular + mocks: `ReferenceError: module not defined` in Client-Integration tests

Till now, I've been trying to get client-unit-test mode running and refreshing properly. So far, it all seems running just fine. Next point on my list was adding urigo:angular and angular:angular-mocks (1.4.2) to see how it works with Meteor and testing.

Setting up a very simple single-file Angular app went fine. I've added some trivial client-unit tests and everything was working just great. Angular-mocks were doing their job well, too. App was refreshing, tests were updated properly. Optimistically, I clicked 'add sample integrations tests'. Finished, ran, green, all great.

However, the sample tests are self-contained. All code is dropped to somewhere in /tests directory, nothing really touches the actual application, and of course the auto-generated sample tests have nothing to do with angular. So, I copied/edited some tests from the client-unit mode and ... boom:

ReferenceError: module is not defined

at:

beforeEach(module('mytestapp'));

It's possible that not only I am seeing such issues. Problem seemed quite obvious: In client-integration, AngularJS (or rather, the mocks) were not loaded or at least not available. Quite strange since everything worked fine in client-unit mode. However, I checked configs, packages, and even the emitted source of the test runner, and everything indicated that the angular-mocks are provided. There had to be some other cause.

AngularMocks expect Jasmine to be already present

AngularMocks 1.4.2 at line 2190 expects window.jasmine or window.mocha to be already defined. Typically, it is set by jasmine-core, but in Meteor, it is the sanjo:jasmine package that ensures that window.jasmine is properly set. Both the expectation and the provision are performed at package-load time.

For all meteor packages, load order are determined by the interpackage dependencies. Since angular:angular-mocks can be used with either Mocha or Jasmine (or probably many other frameworks too), it so it can't/shouldn't strictly depend on either of them. (It seems someone wouldn't want to have angular-mocks-jasmine and angular-mocks-mocha which would probably be the most reasonable way, but, anyways..).

Solution #1: add proper optional dependecy to angular-mocks package - failed

Fortunatelly, Meteor provides a concept of weak dependency, i.e.:

api.use('sanjo:jasmine', ['client'], {weak: true});

which ensures that such package is loaded before the current one - but it is not pulled until something else requires it strongly (weak:false or no specifier at all). That's just perfect here, I wondered, why wasn't that working?

I was suprised to see that despite adding angular:angular-mocks by meteor add, the only actual Meteor packages here are:

urigo:angular
sanjo:jasmine

It turns out that you will not find angular:angular-mocks Meteor package on GitHub, nor in the Meteor core set (well, obvious). It simply is not a Meteor package. I checked packages.meteor.com package directory and it turned out to be ... a Bower package.

..and, as you may see in its bower.json, the only dependency is Angular and there are no weak dependencies. I searched a little and it turns out that Bower has no such concept as "weak/optional dependency".

Too bad, I though, but maybe there is some option for Meteor package to indicate a "reverse weak dependency", that is, to indicate not what packages must be loaded before mine, but rather, what packages my package has to be loaded before. Having such option set on sanjo:jasmine could mark this package as to-be-loaded-before angular:angular-mocks.

Unfortunatelly, this has turned out to be not possible either because Meteor does not support it now, and it doesn't seem planned either. Moreover such feature was requested earlier and seems to have died. Maybe it's worth resurrecting that topic..

Solution #2: wrap angular-mocks into meteor package - ok!

Since Bower does not support optional dependencies and Meteor does, that's pretty obvious attempt. I tried to provide my own 'angular-mocks' with that weak dependency specified:

Package.onUse(function(api) {
  api.versionsFrom('1.1.0.3');
  api.addFiles('client/angular-mocks.js', 'client');

  api.use('sanjo:jasmine', ['client'], {weak: true});
  api.use('urigo:angular', ['client'], {weak: true});
});

meteor remove angular:angular-mocks
meteor add quetzalcoatl:sanjojasmine-angularmocks

It works! However in the long run it seems pointless. I don't want to create and mantain and update such clone just to correctly specify the deps. In case you have such problem and other solutions don't work for you, you can at least temporarily solve it like that.

Solution #3: add dependencies in proper order - ok!

Just for fun, I checked if the order of adding packages to meteor app makes any difference:

meteor add urigo:angular my-mocks sanjo:jasmine                                  # works
meteor add my-mocks urigo:angular sanjo:jasmine                                  # works
meteor add urigo:angular sanjo:jasmine my-mocks                                  # works

meteor add urigo:angular angular:angular-mocks sanjo:jasmine                     # broken
meteor add urigo:angular sanjo:jasmine angular:angular-mocks                     # works
meteor add sanjo:jasmine urigo:angular angular:angular-mocks                     # works

What. Well, ok, that makes some sense. It seems that for unrelated packages (I mean, with no dependencies between each other), Meteor seems to keep them in order of addition. angular-mocks required jasmine to be loaded earlier, and simply adding sanjo:jasmine seems to do the trick.

And since it does not matter if you do that in one line or three:

# will break
meteor add urigo:angular
meteor add angular:angular-mocks
meteor add sanjo:jasmine

# will work
meteor add sanjo:jasmine
meteor add urigo:angular
meteor add angular:angular-mocks

I have to admit that I actually hit that problem because I first created the app, added angular, added mocks, and only after running I noticed I forgot to add sanjo:jasmine.

Therefore, if you have problems with ReferenceError: module not defined when using angular:angular-mocks and sanjo:jasmine, you can simply try:

meteor remove angular:angular-mocks
meteor add angular:angular-mocks

Since you probably already have sanjo:jasmine, then it's enough to make sure angular-mocks have been added after that.

However, since there still is no relation between those two requested packages, Meteor can in fact load them in any order, and this trick doesn't guarantee anything, it can change with different Meteor versions. It just happens to work today.

Solution #4: create a dummy package that will pull mocks in order - ok!

If in the absence of dependencies Meteor doesn't reorder the list of loaded packages, then maybe:

Package.onUse(function(api) {
  api.versionsFrom('1.1.0.3');

  api.use('sanjo:jasmine', ['client'], {weak: true});
  api.use('urigo:angular', ['client'], {weak: true});
  api.use('angular:angular-mocks', ['client']);
});

Note I removed all JS files and just added a dep to angular-mocks. It turns out to be working regardless of the order of adding packages

meteor add urigo:angular my-mocks sanjo:jasmine                                  # works
meteor add my-mocks urigo:angular sanjo:jasmine                                  # works
meteor add urigo:angular sanjo:jasmine my-mocks                                  # works

As version numbers are not pinned on these dependencies, Meteor will select the most current ones from the directory, saving me from mantaining a clone of angular:angular-mocks.

Just as with solution #3, however, since there still is no relation between those two requested packages, Meteor can in fact load them in any order, and this trick doesn't guarantee anything, it can change with different Meteor versions. It just happens to work today.

Once important note to solution #4: since mocks are now implicitly provided by that dummy module, you have to remove angular:angular-mocks or else your app will fetch it on its own and nothing will change.

Afterthoughts

Of course it's disputable which solution is the best.. The best would be #1 but currently it is not possible. Oh, sorry, the very best would be to have angular-mocks don't expect having a jasmine before it or having angular-mocks-jasmine that depends on jasmine, but that's out of the scope for now.

When you hit this exact problem described here, then when picking between solution #3 and #4 you effectively have a choice between (#3):

meteor remove angular:angular-mocks
meteor add angular:angular-mocks

and (#4):

meteor remove angular:angular-mocks
meteor add my-dumy-mocks

and neither of them actually guarantee you much. So, pick #3, least surprise principle.

Similarly, when starting a new project, you have a choice of:

(#3) remember to add the dependencies in correct order
(#4) remember to add my-dummy-mocks instead of angular-mocks

and neither of them actually guarantee you much - but the first has much lower risk of suddenly starting to fail (preserving package-addition-order vs. preserving-order-of-dependencies). So, again, pick #3.

In case someone asked: this is why I am not publishing packages for #1 and #4. I also would currently advise against doing it, but of course there's no way of stopping anyone. If you consider doing it, then please, focus on lobbying for reverse weak dependencies (so sanjo:jasmine can fix that) or lobbying for specialized angular-mocks-jasmine (anyways, why just jasmine and mocha are so special?)

2015-08-28

Windows + meteor + sanjo:jasmine + karma: hot code updates prevent automatic test rerunning

It turns out that last time I was a bit wrong! Upon hot code update, not only PhantomJS jas problems. Whole Karma has, and has them regardless of the selected browser.

Velocity/Sanjo/Karma/(PhantomJS|Chrome|.+) hangs when Meteor auto-rebuilds your project

I must have been tired back then and I didn't notice that autoupdate breaks running tests in Chrome, too. Please see this issue for analysis on that. In short, I've learned that this in fact is Karma problem, or rather, Chokidar@Windows problem: when observed directory is removed or renamed, Chokidar stops observing it - and that's exactly what happens to whole build directory. Fortunatelly, after some work and a bit of hints from Sanjo about (re)starting Karma, I managed to work it around by:

detecting when autoupdate is about to happen
deterministically killing Karma in that case
detecting when autoupdate finishes
restarting Karma right after that

Here's a patch. Yay, now it really actually works with hot updates.

All details are described in the issue, but it's worth pointing out few things:

since Karma dies just before autoupdate begans, all Chokidars file locks are freed - this means no EPERMs during rebuild, and meteor usually can just rename the dirs with no cp-r/rm-r fallbacks
since Karma dies and rises, there's little point in watching any application files at all - change to client files will cause autoupdate (and restart Karma), change to server files will kill&restart whole app (and Karma too). The patch mentioned above turns these watches off, but leaves watches on test/spec files.
Meteor raises two interesting 'events': 'message{refresh:client}' and 'onListening'

Message: refresh-client

reminder: hot code updates are either 'refreshable' or 'not-refreshable'. If you change only 'client' code, then it's the former. Clients will refresh and server won't notice. When you change any 'server' code (/server, /lib, ...), then it's the latter and whole app must be refreshed on both sides. By 'refreshed' I mean kill-everything-and-restart.

refresh:client is broadcasted by the application that is about to start an autoupdate cycle in 'refreshable' mode. Actually, there's a internal listener on that event which starts the autoupdate process. Listening to that event will run your code before the autoupdate starts, but only because that internal listener uses setTimeout(.., 100ms) to delay the process just a bit, so any custom listeners added must act .. quickly or assume running "in parallel" with the build.

During 'non-refreshable' mode does not raise that event, as the app is going to terminate and fully restart in a moment, what will cause all clients to refresh just like during any server reboot.

WebApp.onListening event

Basically, whatever listener is registered there, it's invoked when the hosted application is ready to serve requests. This surely means that any build processes have finished. It is invoked in all three cases:

application's initial start
after refreshable update finishes rebuilding
not-refreshable update (because it's in fact 'initial start' anyways)

However, using this event needs some care. When raised on application's start, it is called before Velocity starts its startup tasks. During refreshable autoupdate it seems to be called last. Here in this case the handler wanted to start Karma after refreshable update, but the same handler is called during app's start when few-second-later the Velocity would start Karma too. That means that handlers may need to check at what stage they were invoked, just to make sure to not race with Velocity at application's start.

Another (small) catch with onListening is that it does not always run when serving starts. If the app is already running, registering handler to that event will not wait for the next update/restart, but instead will immediatelly that handler. Only handlers registered before the app is ready are delayed, stored and executed when its ready.

Another (big) catch is coming along with previous: I really said delayed, stored. Handlers registered when the app is already running are not stored and they are just run immediatelly, and they will not run upon next update/restart. Actually, those handlers registered early will not fire either. It turns out that the list of delayed handlers is cleared after the event occurs. List is cleared as handlers are invoked.

This causes interesting problem: It's tricky to permanently listen to onListening. You have to register the handler at the correct moment of time when the application is not considered to be running or else the handler will immediatelly fire. Fortunatelly, I noticed that one such points of time is the message-refresh-client. This notification is sent when preparing for a quick rebuild, so any onListening registered during message-refresh-client will be delayed and called when refresh cycle ends.

Any examples?

Just in case you missed the link, here's the patch that fixes re-running tests when hot code update happens. It uses these two events to do the job of restarting Karma. Oh, and also some small tweaks to sanjo-karma were needed to be able to actually stop Karma.

2015-08-19

Windows + meteor + sanjo:jasmine: fighting with EPERMs and quest for HiddenChrome

I recently tried using Meteor to build and test a simple application. Since I got used to Jasmine, I tried the mainstream plugin sanjo:jasmine for Velocity bundled with Meteor.

When having some unit-tests, autoupdates cause "Error EPERM, stat" and/or "Error EPERM, unlink"

I have to say, I'm not delighted with testing experience. At first glance it looks great. However during first few days I stumbled upon several bugs related to deleting directory trees in autobuilding/autorefreshing, very annoying, it literally killed (I mean, crashed) the server on each code update and forced me to rerun it manually - which invalidates the whole live/hotswapping idea. Ah yes, I forgot to say I'm on Windows. It probably works perfectly on Linux, where the filesystem behaves differently. Poor me.

I was able to trace and somewhat patch them up, but it doesn't 'feel stable'. I have observed that during autoupdate, even with those patches, the renameDirAlmostAtomically still falls back to cp-r/rm-r fallback, what probably shouldn't have place. Anyways, at least meteor stopped crashing due to EPERMs :)

You can find the rimraf patch here and the meteor patch here

Velocity/Sanjo/Karma by default opens a Chrome window to run the tests

Next thing that bothered me was the Chrome window that was repeatedly opened and closed by Karma on each autoupdate. I understand that this is necessary as the whole app is being killed including Karma and that Karma need to have an actual runner, but damn! when you work on a single monitor work station, you can't even get that Chrome minimizes because it will get soon killed and restarted and a new window will popup, cluttering your workspace again.

Tired by it I searched the web for some solution and I found that it's relatively easy to switch sanjo/jasmine to use PhantomJS instead of Chrome. Great, I knew PhantomJS back from working with Chutzpah. However, despite everybody talking how great is using hidden PhantomJS instead of windowed Chrome, the hints on how to do that varied. The most common were:

set environment var JASMINE_BROWSER=PhantomJS meteor
set environment var JASMINE_BROWSER=PhantomJS meteor run
set environment var JASMINE_BROWSER=PhantomJS meteor --test
set environment var JASMINE_BROWSER=Chrome PhantomJS meteor run (sic! what a pity I didn't save the link where I saw that)

I wouldn't be writing about it if any of those worked. None was. After looking at the sources I noticed that this is almost directly passed to Karma, and that:

this option takes the keyword like 'Chrome' or 'PhantomJS' only, no "commands" like meteor --test are supported
this options does not support specifying multiple browsers, even though Karma could

So, at the time of writing, sanjo:jasmine ver 0.17.0, the only proper way of setting this variable is:

JASMINE_BROWSER=Chrome
JASMINE_BROWSER=PhantomJS

otherwise it won't run anything, as sanjo:jasmine will not be able to pick a 'launcher' (karma-chrome-launcher, karma-phantomjs-launcher, etc) and Karma will start with no runners (so i.e. you will still be able to manually open localhost:9876 in some Chrome window).

So far so good. I stopped meteor from crashing, I switched it to PhantomJS, got rid of Chrome window! Did I mention that instead of a Chrome window I got now an empty console window?

Velocity/Sanjo/Karma/PhantomJS by default opens a console window to run the tests

I started to feel a little grumpy about that, but I kept digging and I've found that somewhere in sanjo's long-running-child-process there's a series of child_process.spawns that run a spawnscript with stdio bound to logfile, and that script in turn creates child PhantomJS process. That 'spawnscript' had an option detach: true set. That comes from NodeJS and is probably important on Linux to 'daemonize' a child, but on Windows the parent processes does not wait for their children anyways, there's no point in that option. I removed that option and wow, console window disappeared. Now I had the test process truly hidden.

You can find the patch here

Velocity/Sanjo/Karma/PhantomJS hangs when Meteor auto-rebuilds your project

How great was my surprise and disbelief when I noticed that the Velocity html reported stopped refreshing the test results ..but only sometimes:

1a. (re)started my meteor app, page got autoupdated, all tests were run
1b. edited specs, page was not updated (good!), testrunner noticed it, tests ran, results displayed (cool!)
1c. edited apps code, page was autoupdated (goodb!) ..but tests were **not** re-ran, old results displayed (wha?)

2a. (re)started my meteor app, page got autoupdated, all tests were run (ok, so nothing broken)
2b. edited apps code, page was updated (good!) ..but tests **not** ran, old results displayed (see the pattern)

3a. (re)started my meteor app, page got autoupdated, all tests were run, everything fine
3b. edited specs - everything ok, tests ran
3c. edited specs - everything ok, tests ran
3d. edited specs - everything ok, tests ran
3e. edited apps code - app updated, tests **not** ran (yup)
3f. edited apps code - app updated, tests **not** ran
3g. edited specs - everything ok, tests **not** ran (what?!)
3h. edited specs - everything ok, tests **not** ran (...)

4a. (re)started my meteor app, page got autoupdated, all tests were run, everything fine

Noticeable time later wasted on tracing the problem proved that it happens only when using PhantomJS and only when the change to the application code causes restart. I'm pretty sure it's not a fault of PhantomJS. In .meteor/local/log/jasmine-client-unit.log I've found some errors regarding EPERMs and startup crashes mentioning that Package is not defined. Seems that PhantomJS was unable to reload some of the files, maybe there were not properly deleted or properly replaced. I wasn't able to trace it yet.

As I said, I noticed this problem occurrs ~~only~~^*) when PhantomJS is used. When Chrome is used with its irritating window popping up, ~~everything works fine~~^*). I switched back to using Chrome and focused on hiding the window. Unfortunatelly my time was getting shorter and shorter. I tried to find the place when Karma actually starts the child process so I could use some old tricks to send a hide-window message to it, but first I've found karma-chrome-launcher module that finds the Chrome executable and builds the command line options. Hey, maybe Chrome has some cool commandline switch to hide the window?

^*) it turned out to be completely not true. see next post on that

Hiding the Chrome window

After some searching, I learned that most opinions tell it's not possible and that I have to use some third-party tools to send a message to its window to hide it (yeah, echo!). I ignored that and searched for the list of Chrome's commandline options. Here it is in raw form and here it is in a nice table.

As you can see there, there are some promising flags!

// Does not automatically open a browser window on startup (used when
// launching Chrome for the purpose of hosting background apps).
const char kNoStartupWindow[]               = "no-startup-window";

// Causes Chrome to launch without opening any windows by default. Useful if
// one wishes to use Chrome as an ash server.
const char kSilentLaunch[]                  = "silent-launch";

I managed to successfuly run Chrome with --no-startup-window and indeed it launched without any windows. It looked like it launched properly, it spawned all typical children, but the website I tried to make it load inside didn't seem to be actually visited. It maybe possible that this headless mode is only for running apps and not for visiting sites headless*), but it looks very promising as the normal worker tree is set up, just no windows. Yet my time was getting even shorter (probably the counter already has rolled into negative values) so I have not investigated further.

The second option --silent-launch made chrome process very silent. I didn't notice any children spawned at all and the process exited promptly. I doubt it'll be usable for this case.

After I failed my attempts with these options, I turned to less sophisticated ways. On the bottom of the list there are two options:

// Specify the initial window position: --window-position=x,y
const char kWindowPosition[]                = "window-position";

// Specify the initial window size: --window-size=w,h
const char kWindowSize[]                    = "window-size";

I edited karma-chrome-launcher\index.js to include options to move it completely out of the working area:

return [
  '--user-data-dir=' + this._tempDir,
  '--no-default-browser-check',
  '--no-first-run',
  '--disable-default-apps',
  '--disable-popup-blocking',
  '--disable-translate',
  '--window-position=-800,0',    // <-- added
  '--window-size=800,600'        // <-- added
].concat(flags, [url])

and sure it's no true headless, but still the window is finally out of my sight and tests are properly re-ran whenever I edit either the app or specs. Yeay! I'll probably push a change proposal so those options can be turned on/off on demand.

Solution revisited

After searching a little more, I found out that karma already has a way of passing custom parameters to its browsers, and thanks to it the change was narrowed and moved from karma-chrome-launcher to sanjo:jasmine that was building the karma configuration file.

You can find the patch here and if using that, don't forget to remove JASMINE_BROSER env variable, or set it to HiddenChrome:

JASMINE_BROWSER=HiddenChrome

Happy coding with new "HiddenChrome" browser launcher.

final notes

It feels dirty though. I'd prefer really-hiding that window and/or using PhantomJS.

..and that was only the client-unit tests. client-integration, server-integration and server-unit yet to be tried.. I mean, if server-unit module gets fixed, it's officially said to be unstable now and it's advised to use server-integration mode instead to run them.

2015-01-09

Quick reminder about AttachedProperties and AttachedBehaviors

Yet again, I produced too much text than I intended for a short response on StackOverflow.. If somebody is interested in, the question came from StackOverflow: Dynamic Conditional Formatting in WPF.

Let's start with WPF support for "Attached properties". You can literally attach them to anything you like. A Grid.Row is example of such attached property.

Once you create a couple of your att.props, you observe and bind them, and also bind to them as normal:

<Foo x:Name="foo" myns:MyExtension.MyAttachedProperty1="SomeSource.AProperty" />
<Bar myns:MyExtension.MyAttachedProperty2="{Binding Path=(myns:MyExtension.MyAttachedProperty1).Width, ElementName=foo }" />

note that when binding to an attached property, you must use parenthesis in the property path, or else the MyExtension class name will be treated as a source instead of as a classname-prefix. You may also need namespace prefixes (xmlns:my=... + my: everywhere)

Once you learn/master attached properties, things start to be fun! Since you can attach them and bind on almost anything, you can introduce smart extensions like:

<TextBlock m:MyEx.UnitSystem="{Binding ..}" m:MyEx.SourceValue="{Binding ..}" />

Note that TextBlock.Text is not bound. The idea behind it is that SourceValue attached property gets the raw value to be displayed, and its change handlers observe the changes to both DisplayedValue and UnitSystem, and they translate the values and set the Text on the component. Not much suprising.

But the suprising fact that is easily omitted is, that your code will be totally decoupled. You will have a source-of-value, source-of-unitsystem, both just pulled from datasource. Your calculations will just emit the value. And yet another class/file will define the plugin that handles the conversions and updates the TextBlock.

But, ok, so we have the source-value bindings attached like above. Where to put the actual code that handles the changes? Of course, you can put that right into these attached dependency properties, just attach uimetadata with a change-handler and done. Since every change-handler receives the originating DependencyObject=TextBlock, those change handlers will be able to update the Text.

But it'll get really messy once you need to observe two, three or more source properties, because every one of those will need to be really carefully tracked.

So, here's what I like to do in such cases:

<TextBlock my:MyEx.UnitSystem="{Binding ..}"
           my:MyEx.SourceValue="{Binding ..}"
           Text="{Binding (my:MyEx.DisplayedValue), RelativeSource={RelativeSource Self}}">
    <my:AutoUnitConversion />
</TextBlock>

The AutoUnitConversion is an AttachedBehavior that after attaching to TextBox observes changes to UnitSystem and SourceValue and calculates the displayed value. The behavior could directly set the Text of the TextBlock, but then, it would be usable only with TextBlocks. So, instead, I'd make it emit the calculated value as a third property, DisplayedValue. Note the usage of RelativeSource=self, since the output attached property is set right on the parent component. In this setup, you may then directly reuse the AttachedBehavior in other places - just the final Text/Value/etc binding will change.

I think you see now how powerful it can be. It's more complex than styles, bindings and triggers, but on the other hand it allows you to literally attach any logic and any behavior. While bindings and conditions give you some sort of a language to set the rules, it's sometimes simply not enough, or sometimes it actually gets overcomplicated in XAML and writing the same logic by C# or VB is much simpler than via multibindings, converters and conditions. In such case, Behaviors are great!

..but this hint would not be complete without mentioning that Behaviors are really more complex, as you often need to set up or observe some bindings from the Code, not XAML, and also you need to remember about Attach/Detach lifecycle, and also they put some additional overhead to whole system as they introduce more properties/values to be tracked by WPF. It's really hard to tell if it overhead is higher or lower than MultiBindings and lots of Converters. I'd guess it's actually lower, but I have not measured that.

By the way, I almost forgot to add the link: here's a very quick overview about AttachedBehaviors. Please note that this term is used for two things: 'quick&dirty' behaviors registerd through setting a AttachedDependencyProperty, and 'fully-fledged' behaviors registered by XAML tags ("Blend Behaviors", named after MS ExpressionBlend). They look a bit different, but the idea of operation is mostly the same. While creating and using the latter ones is a bit more involving, they tend to be structured better, easier&more reusable than the former, and they come with handy Attach/Detach methods. It's good to familiarize yourself with both types!

2015-01-08

Random findings about EF and the way it sets up Connections and Contexts

Here's the long nontrimmed version of my answer, with all the babbling. If somebody is interested in, the question came from StackOverflow: Dynamic Connection String in Entity Framework.

Ok, as I have not found any trace of the code I produced back then, I started digging again, trying to remember what I tried doing. I remember that finally everything almost worked but I got stuck at ... not being able to provide nor build an instance of EntityConnection because its connection string required three mapping files and as I used Code-First I simply didn't have them. However, in your case it should do the trick.

So.. the most important thing is of course the InternalContext. It's the brain here and it can be initialized in many ways, drived by various DbContext constructors. I first targetted the "ObjectContext" overload and tried to provide my own already-connected instance, but that proved impossible as I finally managed to get hold on a service that returns a provider that returns a factory that builds ObjectContexts ... but this factory required me to pass a DbContext. Remember that I wanted to have ObjCtx to be able to create call DbCtx ctor's overload.. Zonk.

I decompiled the factory and yet again it turned out that current implementation of DbCtx/ObjCtx and this factory that the only point where you can actually callable is inside the DbContext's constructors. The ObjectContext produced by the factory gets bound to that specific DbCtx and it's not possible to untangle them later. I remember that it's all about Connections and Metadata. It was something like "DbCtx provides connection", "DbCtx delegates all other jobs to ObjCtx", "ObjCtx provides metadata" but "ObjCtx does not know where to get metadata from", so it asks a 'service' to find it, which in turn looks up for the connection, which it gets from DbCtx..

There was also some play with IObjectContextAdapter, but I don't remember it now, except for the fact that I got to a point where I rolled in my own ObjectContext2 class (I think I just subclassed the ObjectContext) which provided everything all by itself and didn't need that factory, but I got stuck at not being able to manually construct and initialize Metadata(Workspace?) properly.

So, that was the end of "EagerInternalConnection" path, which was the most promising one, since it did not rely on any defaults and just took everything I provided to it.

The last thing left was InitializeLazyInternalContext (you mentioned), which basically initializes everything according to defaults. That's either the true defaults related to the name of the context class, or you can provide a "nameOrConnectionString" (which I could not use since it expects EntityConnectionString and I used Code-First). Or, there's third option that takes DbCompiledModel. It's protected but when you inherit from DbContext you can call it easily.

It's a bit tricky to get an instance of DbCompiledModel though. It can't be built directly, it has to be obtained from DbModelBuilder, which in turn is often only temporarily available during your XYZContext initialization. Surely you remember OnModelCreating(DbModelBuilder) method where all model configurations take place. So.. but it's inside XYZContext and you need the CompiledModel before you start construction your XYZContext/DbContext again. You can refactor all the model-setup out of the context, or you may simply hack in and expose the protected OnModelCreating method even statically to be able to build models manually. However, there are few bits missing in the OnModelCreating method. There are some base policies and conventions that are set before this method is called, so if you'll be creating them manually.. you'd need to again decompile&find the part that sets them, and I remember it's well private/hidden so you'd need to invoke it via reflection or copy all the setup the code..

Anyways, after refactoring some code I managed to have the DbModelBuilder configured, model build, compiled, passed to new MyDbModel constructor -- and it gridnded to halt again, because of some issues I dont recall now, Sorry..

The factories/services I talked about. It's all around IDbDependencyResolver interface. Look at System.Data.Entity.Infrastructure.DependencyResolution. You'll find it inside. There's handy implementation of a service/instace resolver called SingletonDependencyResolver<T> that will be sufficient in most cases.

Registering resolvers is tricky. If I remember well, I injected the resolvers through (same namespace)DbConfigurationManager.(static)Instance.GetConfiguration() which returns the IoC configuration object and .AddDependencyResolver(..) or .RegisterSingleton<TService>. All internal though, so must use reflection here. I remember that the IoC and resolvers can also be configured with special entries in app.config settings, but I didn't use it as I was focused at 100% runtime configuration. Now I see that actually I didn't need that, when I wanted to inject the resolvers I could certainly do that from appconfig, as the resolver classes would not change in the runtime. You may try so instead of reflection.

Ok, so the resovlers.. There are a bunch of them you can override. I remember playing with ProviderFactoryResolver, InvariantNameResolver, and ProviderServiceResolver. The first one is meant to provide DbProviderFactory that is the root of all connection builders. The second one extracts 'invariant name' from the factory - that's in fact the "dialect" name like "sql2008". Many things are keyed/identified by this "invariant name". I don't remember much about the third ProviderServiceResolver one.

Look first at DbProviderFactoryResolver. If you register your own resolver if this type, you will be able to inject your own DbProviderFactory-ies. Your own DbProviderFactory seems crucial. Look at decompiled EntityConnection and its ChangeConnection method. It is one of the core methods that, well, sets or changes the connection string.

private void ChangeConnection(string ..blah..) {
    ...
    DbProviderFactory fac = null;
    ...
    fac = DbDependencyResolverExtensions.GetService<DbProviderFactory> ( ... )
    ...
    connection = EntityConnection.GetStoreConnection(fac);
    ...
    connstr = ..blah..["provider connection string"];
    DbInterception.Dispatch.Connection.SetConnection(connection, new DbConnectionPropertyInterceptionContext<string>(..).WithValue(connstr) );
}

Yep, the DbProviderFactory you see there is just it. Its CreateConnection returns a DbConnection (it can be yours) and then it configured with 'provider connection string' right from the settings from the entityconnectionstring section. So, if you additionally roll in your own DbConnection, build by your own DbProviderFactory, it could then ignore the incoming connstring in favor of whatever you want.

Also, I have not investigated the DbInterception.Dispatch.Connection.SetConnection part because back then my play-time has run out. There is some slight chance that Interception is mean literally and that you will be able to "just register" an interceptor that will override the "SetConnectionString".

But, well.. that's enormous amount of work. Really. Using separate file to exploit partial to expose additional base DbContext constructor and then inheriting from the autogenerated context class and providing the connstring through that is much much simpler!