meteor + sanjo:jasmine + urigo:angular + mocks: `ReferenceError: module not defined` in Client-Integration tests

Till now, I've been trying to get client-unit-test mode running and refreshing properly. So far, it all seems running just fine. Next point on my list was adding urigo:angular and angular:angular-mocks (1.4.2) to see how it works with Meteor and testing.

Setting up a very simple single-file Angular app went fine. I've added some trivial client-unit tests and everything was working just great. Angular-mocks were doing their job well, too. App was refreshing, tests were updated properly. Optimistically, I clicked 'add sample integrations tests'. Finished, ran, green, all great.

However, the sample tests are self-contained. All code is dropped to somewhere in /tests directory, nothing really touches the actual application, and of course the auto-generated sample tests have nothing to do with angular. So, I copied/edited some tests from the client-unit mode and ... boom:

ReferenceError: module is not defined

at:

beforeEach(module('mytestapp'));

It's possible that not only I am seeing such issues. Problem seemed quite obvious: In client-integration, AngularJS (or rather, the mocks) were not loaded or at least not available. Quite strange since everything worked fine in client-unit mode. However, I checked configs, packages, and even the emitted source of the test runner, and everything indicated that the angular-mocks are provided. There had to be some other cause.

AngularMocks expect Jasmine to be already present

AngularMocks 1.4.2 at line 2190 expects window.jasmine or window.mocha to be already defined. Typically, it is set by jasmine-core, but in Meteor, it is the sanjo:jasmine package that ensures that window.jasmine is properly set. Both the expectation and the provision are performed at package-load time.

For all meteor packages, load order are determined by the interpackage dependencies. Since angular:angular-mocks can be used with either Mocha or Jasmine (or probably many other frameworks too), it so it can't/shouldn't strictly depend on either of them. (It seems someone wouldn't want to have angular-mocks-jasmine and angular-mocks-mocha which would probably be the most reasonable way, but, anyways..).

Solution #1: add proper optional dependecy to angular-mocks package - failed

Fortunatelly, Meteor provides a concept of weak dependency, i.e.:

api.use('sanjo:jasmine', ['client'], {weak: true});

which ensures that such package is loaded before the current one - but it is not pulled until something else requires it strongly (weak:false or no specifier at all). That's just perfect here, I wondered, why wasn't that working?

I was suprised to see that despite adding angular:angular-mocks by meteor add, the only actual Meteor packages here are:

• urigo:angular
• sanjo:jasmine

It turns out that you will not find angular:angular-mocks Meteor package on GitHub, nor in the Meteor core set (well, obvious). It simply is not a Meteor package. I checked packages.meteor.com package directory and it turned out to be ... a Bower package.

..and, as you may see in its bower.json, the only dependency is Angular and there are no weak dependencies. I searched a little and it turns out that Bower has no such concept as "weak/optional dependency".

Too bad, I though, but maybe there is some option for Meteor package to indicate a "reverse weak dependency", that is, to indicate not what packages must be loaded before mine, but rather, what packages my package has to be loaded before. Having such option set on sanjo:jasmine could mark this package as to-be-loaded-before angular:angular-mocks.

Unfortunatelly, this has turned out to be not possible either because Meteor does not support it now, and it doesn't seem planned either. Moreover such feature was requested earlier and seems to have died. Maybe it's worth resurrecting that topic..

Solution #2: wrap angular-mocks into meteor package - ok!

Since Bower does not support optional dependencies and Meteor does, that's pretty obvious attempt. I tried to provide my own 'angular-mocks' with that weak dependency specified:

Package.onUse(function(api) {
  api.versionsFrom('1.1.0.3');
  api.addFiles('client/angular-mocks.js', 'client');

  api.use('sanjo:jasmine', ['client'], {weak: true});
  api.use('urigo:angular', ['client'], {weak: true});
});

meteor remove angular:angular-mocks
meteor add quetzalcoatl:sanjojasmine-angularmocks

It works! However in the long run it seems pointless. I don't want to create and mantain and update such clone just to correctly specify the deps. In case you have such problem and other solutions don't work for you, you can at least temporarily solve it like that.

Solution #3: add dependencies in proper order - ok!

Just for fun, I checked if the order of adding packages to meteor app makes any difference:

meteor add urigo:angular my-mocks sanjo:jasmine                                  # works
meteor add my-mocks urigo:angular sanjo:jasmine                                  # works
meteor add urigo:angular sanjo:jasmine my-mocks                                  # works

meteor add urigo:angular angular:angular-mocks sanjo:jasmine                     # broken
meteor add urigo:angular sanjo:jasmine angular:angular-mocks                     # works
meteor add sanjo:jasmine urigo:angular angular:angular-mocks                     # works

What. Well, ok, that makes some sense. It seems that for unrelated packages (I mean, with no dependencies between each other), Meteor seems to keep them in order of addition. angular-mocks required jasmine to be loaded earlier, and simply adding sanjo:jasmine seems to do the trick.

And since it does not matter if you do that in one line or three:

# will break
meteor add urigo:angular
meteor add angular:angular-mocks
meteor add sanjo:jasmine

# will work
meteor add sanjo:jasmine
meteor add urigo:angular
meteor add angular:angular-mocks

I have to admit that I actually hit that problem because I first created the app, added angular, added mocks, and only after running I noticed I forgot to add sanjo:jasmine.

Therefore, if you have problems with ReferenceError: module not defined when using angular:angular-mocks and sanjo:jasmine, you can simply try:

meteor remove angular:angular-mocks
meteor add angular:angular-mocks

Since you probably already have sanjo:jasmine, then it's enough to make sure angular-mocks have been added after that.

However, since there still is no relation between those two requested packages, Meteor can in fact load them in any order, and this trick doesn't guarantee anything, it can change with different Meteor versions. It just happens to work today.

Solution #4: create a dummy package that will pull mocks in order - ok!

If in the absence of dependencies Meteor doesn't reorder the list of loaded packages, then maybe:

Package.onUse(function(api) {
  api.versionsFrom('1.1.0.3');

  api.use('sanjo:jasmine', ['client'], {weak: true});
  api.use('urigo:angular', ['client'], {weak: true});
  api.use('angular:angular-mocks', ['client']);
});

Note I removed all JS files and just added a dep to angular-mocks. It turns out to be working regardless of the order of adding packages

meteor add urigo:angular my-mocks sanjo:jasmine                                  # works
meteor add my-mocks urigo:angular sanjo:jasmine                                  # works
meteor add urigo:angular sanjo:jasmine my-mocks                                  # works

As version numbers are not pinned on these dependencies, Meteor will select the most current ones from the directory, saving me from mantaining a clone of angular:angular-mocks.

Just as with solution #3, however, since there still is no relation between those two requested packages, Meteor can in fact load them in any order, and this trick doesn't guarantee anything, it can change with different Meteor versions. It just happens to work today.

Once important note to solution #4: since mocks are now implicitly provided by that dummy module, you have to remove angular:angular-mocks or else your app will fetch it on its own and nothing will change.

Afterthoughts

Of course it's disputable which solution is the best.. The best would be #1 but currently it is not possible. Oh, sorry, the very best would be to have angular-mocks don't expect having a jasmine before it or having angular-mocks-jasmine that depends on jasmine, but that's out of the scope for now.

When you hit this exact problem described here, then when picking between solution #3 and #4 you effectively have a choice between (#3):

meteor remove angular:angular-mocks
meteor add angular:angular-mocks

and (#4):

meteor remove angular:angular-mocks
meteor add my-dumy-mocks

and neither of them actually guarantee you much. So, pick #3, least surprise principle.

Similarly, when starting a new project, you have a choice of:

(#3) remember to add the dependencies in correct order
(#4) remember to add my-dummy-mocks instead of angular-mocks

and neither of them actually guarantee you much - but the first has much lower risk of suddenly starting to fail (preserving package-addition-order vs. preserving-order-of-dependencies). So, again, pick #3.

In case someone asked: this is why I am not publishing packages for #1 and #4. I also would currently advise against doing it, but of course there's no way of stopping anyone. If you consider doing it, then please, focus on lobbying for reverse weak dependencies (so sanjo:jasmine can fix that) or lobbying for specialized angular-mocks-jasmine (anyways, why just jasmine and mocha are so special?)

Windows + meteor + sanjo:jasmine + karma: hot code updates prevent automatic test rerunning

It turns out that last time I was a bit wrong! Upon hot code update, not only PhantomJS jas problems. Whole Karma has, and has them regardless of the selected browser.

Velocity/Sanjo/Karma/(PhantomJS|Chrome|.+) hangs when Meteor auto-rebuilds your project

I must have been tired back then and I didn't notice that autoupdate breaks running tests in Chrome, too. Please see this issue for analysis on that. In short, I've learned that this in fact is Karma problem, or rather, Chokidar@Windows problem: when observed directory is removed or renamed, Chokidar stops observing it - and that's exactly what happens to whole build directory. Fortunatelly, after some work and a bit of hints from Sanjo about (re)starting Karma, I managed to work it around by:

• detecting when autoupdate is about to happen
• deterministically killing Karma in that case
• detecting when autoupdate finishes
• restarting Karma right after that

Here's a patch. Yay, now it really actually works with hot updates.

All details are described in the issue, but it's worth pointing out few things:

• since Karma dies just before autoupdate begans, all Chokidars file locks are freed - this means no EPERMs during rebuild, and meteor usually can just rename the dirs with no cp-r/rm-r fallbacks
• since Karma dies and rises, there's little point in watching any application files at all - change to client files will cause autoupdate (and restart Karma), change to server files will kill&restart whole app (and Karma too). The patch mentioned above turns these watches off, but leaves watches on test/spec files.
• Meteor raises two interesting 'events': 'message{refresh:client}' and 'onListening'

Message: refresh-client

reminder: hot code updates are either 'refreshable' or 'not-refreshable'. If you change only 'client' code, then it's the former. Clients will refresh and server won't notice. When you change any 'server' code (/server, /lib, ...), then it's the latter and whole app must be refreshed on both sides. By 'refreshed' I mean kill-everything-and-restart.

refresh:client is broadcasted by the application that is about to start an autoupdate cycle in 'refreshable' mode. Actually, there's a internal listener on that event which starts the autoupdate process. Listening to that event will run your code before the autoupdate starts, but only because that internal listener uses setTimeout(.., 100ms) to delay the process just a bit, so any custom listeners added must act .. quickly or assume running "in parallel" with the build.

During 'non-refreshable' mode does not raise that event, as the app is going to terminate and fully restart in a moment, what will cause all clients to refresh just like during any server reboot.

WebApp.onListening event

Basically, whatever listener is registered there, it's invoked when the hosted application is ready to serve requests. This surely means that any build processes have finished. It is invoked in all three cases:

• application's initial start
• after refreshable update finishes rebuilding
• not-refreshable update (because it's in fact 'initial start' anyways)

However, using this event needs some care. When raised on application's start, it is called before Velocity starts its startup tasks. During refreshable autoupdate it seems to be called last. Here in this case the handler wanted to start Karma after refreshable update, but the same handler is called during app's start when few-second-later the Velocity would start Karma too. That means that handlers may need to check at what stage they were invoked, just to make sure to not race with Velocity at application's start.

Another (small) catch with onListening is that it does not always run when serving starts. If the app is already running, registering handler to that event will not wait for the next update/restart, but instead will immediatelly that handler. Only handlers registered before the app is ready are delayed, stored and executed when its ready.

Another (big) catch is coming along with previous: I really said delayed, stored. Handlers registered when the app is already running are not stored and they are just run immediatelly, and they will not run upon next update/restart. Actually, those handlers registered early will not fire either. It turns out that the list of delayed handlers is cleared after the event occurs. List is cleared as handlers are invoked.

This causes interesting problem: It's tricky to permanently listen to onListening. You have to register the handler at the correct moment of time when the application is not considered to be running or else the handler will immediatelly fire. Fortunatelly, I noticed that one such points of time is the message-refresh-client. This notification is sent when preparing for a quick rebuild, so any onListening registered during message-refresh-client will be delayed and called when refresh cycle ends.

Any examples?

Just in case you missed the link, here's the patch that fixes re-running tests when hot code update happens. It uses these two events to do the job of restarting Karma. Oh, and also some small tweaks to sanjo-karma were needed to be able to actually stop Karma.

Windows + meteor + sanjo:jasmine: fighting with EPERMs and quest for HiddenChrome

I recently tried using Meteor to build and test a simple application. Since I got used to Jasmine, I tried the mainstream plugin sanjo:jasmine for Velocity bundled with Meteor.

When having some unit-tests, autoupdates cause "Error EPERM, stat" and/or "Error EPERM, unlink"

I have to say, I'm not delighted with testing experience. At first glance it looks great. However during first few days I stumbled upon several bugs related to deleting directory trees in autobuilding/autorefreshing, very annoying, it literally killed (I mean, crashed) the server on each code update and forced me to rerun it manually - which invalidates the whole live/hotswapping idea. Ah yes, I forgot to say I'm on Windows. It probably works perfectly on Linux, where the filesystem behaves differently. Poor me.

I was able to trace and somewhat patch them up, but it doesn't 'feel stable'. I have observed that during autoupdate, even with those patches, the renameDirAlmostAtomically still falls back to cp-r/rm-r fallback, what probably shouldn't have place. Anyways, at least meteor stopped crashing due to EPERMs :)

You can find the rimraf patch here and the meteor patch here

Velocity/Sanjo/Karma by default opens a Chrome window to run the tests

Next thing that bothered me was the Chrome window that was repeatedly opened and closed by Karma on each autoupdate. I understand that this is necessary as the whole app is being killed including Karma and that Karma need to have an actual runner, but damn! when you work on a single monitor work station, you can't even get that Chrome minimizes because it will get soon killed and restarted and a new window will popup, cluttering your workspace again.

Tired by it I searched the web for some solution and I found that it's relatively easy to switch sanjo/jasmine to use PhantomJS instead of Chrome. Great, I knew PhantomJS back from working with Chutzpah. However, despite everybody talking how great is using hidden PhantomJS instead of windowed Chrome, the hints on how to do that varied. The most common were:

• set environment var JASMINE_BROWSER=PhantomJS meteor
• set environment var JASMINE_BROWSER=PhantomJS meteor run
• set environment var JASMINE_BROWSER=PhantomJS meteor --test
• set environment var JASMINE_BROWSER=Chrome PhantomJS meteor run (sic! what a pity I didn't save the link where I saw that)

I wouldn't be writing about it if any of those worked. None was. After looking at the sources I noticed that this is almost directly passed to Karma, and that:

• this option takes the keyword like 'Chrome' or 'PhantomJS' only, no "commands" like meteor --test are supported
• this options does not support specifying multiple browsers, even though Karma could

So, at the time of writing, sanjo:jasmine ver 0.17.0, the only proper way of setting this variable is:

JASMINE_BROWSER=Chrome
JASMINE_BROWSER=PhantomJS

otherwise it won't run anything, as sanjo:jasmine will not be able to pick a 'launcher' (karma-chrome-launcher, karma-phantomjs-launcher, etc) and Karma will start with no runners (so i.e. you will still be able to manually open localhost:9876 in some Chrome window).

So far so good. I stopped meteor from crashing, I switched it to PhantomJS, got rid of Chrome window! Did I mention that instead of a Chrome window I got now an empty console window?

Velocity/Sanjo/Karma/PhantomJS by default opens a console window to run the tests

I started to feel a little grumpy about that, but I kept digging and I've found that somewhere in sanjo's long-running-child-process there's a series of child_process.spawns that run a spawnscript with stdio bound to logfile, and that script in turn creates child PhantomJS process. That 'spawnscript' had an option detach: true set. That comes from NodeJS and is probably important on Linux to 'daemonize' a child, but on Windows the parent processes does not wait for their children anyways, there's no point in that option. I removed that option and wow, console window disappeared. Now I had the test process truly hidden.

You can find the patch here

Velocity/Sanjo/Karma/PhantomJS hangs when Meteor auto-rebuilds your project

How great was my surprise and disbelief when I noticed that the Velocity html reported stopped refreshing the test results ..but only sometimes:

1a. (re)started my meteor app, page got autoupdated, all tests were run
1b. edited specs, page was not updated (good!), testrunner noticed it, tests ran, results displayed (cool!)
1c. edited apps code, page was autoupdated (goodb!) ..but tests were **not** re-ran, old results displayed (wha?)

2a. (re)started my meteor app, page got autoupdated, all tests were run (ok, so nothing broken)
2b. edited apps code, page was updated (good!) ..but tests **not** ran, old results displayed (see the pattern)

3a. (re)started my meteor app, page got autoupdated, all tests were run, everything fine
3b. edited specs - everything ok, tests ran
3c. edited specs - everything ok, tests ran
3d. edited specs - everything ok, tests ran
3e. edited apps code - app updated, tests **not** ran (yup)
3f. edited apps code - app updated, tests **not** ran
3g. edited specs - everything ok, tests **not** ran (what?!)
3h. edited specs - everything ok, tests **not** ran (...)

4a. (re)started my meteor app, page got autoupdated, all tests were run, everything fine

Noticeable time later wasted on tracing the problem proved that it happens only when using PhantomJS and only when the change to the application code causes restart. I'm pretty sure it's not a fault of PhantomJS. In .meteor/local/log/jasmine-client-unit.log I've found some errors regarding EPERMs and startup crashes mentioning that Package is not defined. Seems that PhantomJS was unable to reload some of the files, maybe there were not properly deleted or properly replaced. I wasn't able to trace it yet.

As I said, I noticed this problem occurrs only^*) when PhantomJS is used. When Chrome is used with its irritating window popping up, everything works fine^*). I switched back to using Chrome and focused on hiding the window. Unfortunatelly my time was getting shorter and shorter. I tried to find the place when Karma actually starts the child process so I could use some old tricks to send a hide-window message to it, but first I've found karma-chrome-launcher module that finds the Chrome executable and builds the command line options. Hey, maybe Chrome has some cool commandline switch to hide the window?

^*) it turned out to be completely not true. see next post on that

Hiding the Chrome window

After some searching, I learned that most opinions tell it's not possible and that I have to use some third-party tools to send a message to its window to hide it (yeah, echo!). I ignored that and searched for the list of Chrome's commandline options. Here it is in raw form and here it is in a nice table.

As you can see there, there are some promising flags!

// Does not automatically open a browser window on startup (used when
// launching Chrome for the purpose of hosting background apps).
const char kNoStartupWindow[]               = "no-startup-window";

// Causes Chrome to launch without opening any windows by default. Useful if
// one wishes to use Chrome as an ash server.
const char kSilentLaunch[]                  = "silent-launch";

I managed to successfuly run Chrome with --no-startup-window and indeed it launched without any windows. It looked like it launched properly, it spawned all typical children, but the website I tried to make it load inside didn't seem to be actually visited. It maybe possible that this headless mode is only for running apps and not for visiting sites headless*), but it looks very promising as the normal worker tree is set up, just no windows. Yet my time was getting even shorter (probably the counter already has rolled into negative values) so I have not investigated further.

The second option --silent-launch made chrome process very silent. I didn't notice any children spawned at all and the process exited promptly. I doubt it'll be usable for this case.

After I failed my attempts with these options, I turned to less sophisticated ways. On the bottom of the list there are two options:

// Specify the initial window position: --window-position=x,y
const char kWindowPosition[]                = "window-position";

// Specify the initial window size: --window-size=w,h
const char kWindowSize[]                    = "window-size";

I edited karma-chrome-launcher\index.js to include options to move it completely out of the working area:

return [
  '--user-data-dir=' + this._tempDir,
  '--no-default-browser-check',
  '--no-first-run',
  '--disable-default-apps',
  '--disable-popup-blocking',
  '--disable-translate',
  '--window-position=-800,0',    // <-- added
  '--window-size=800,600'        // <-- added
].concat(flags, [url])

and sure it's no true headless, but still the window is finally out of my sight and tests are properly re-ran whenever I edit either the app or specs. Yeay! I'll probably push a change proposal so those options can be turned on/off on demand.

Solution revisited

After searching a little more, I found out that karma already has a way of passing custom parameters to its browsers, and thanks to it the change was narrowed and moved from karma-chrome-launcher to sanjo:jasmine that was building the karma configuration file.

You can find the patch here and if using that, don't forget to remove JASMINE_BROSER env variable, or set it to HiddenChrome:

JASMINE_BROWSER=HiddenChrome

Happy coding with new "HiddenChrome" browser launcher.

final notes

It feels dirty though. I'd prefer really-hiding that window and/or using PhantomJS.

..and that was only the client-unit tests. client-integration, server-integration and server-unit yet to be tried.. I mean, if server-unit module gets fixed, it's officially said to be unstable now and it's advised to use server-integration mode instead to run them.