start/stop DeepaMehta from command line

[jri]

You can start DM from the command line.

With the binary distribution run:

./deepamehta-linux.sh

(or the respective script for your platform)

When you've build from source run:

cd deepamehta
mvn pax:run

Does it work for you?

[jri]

In both cases DM is stopped by typing (in the gogo shell):

stop 0

[JuergeN]

And is there any way to stop DeepaMehta from the command line (outside the gogo shell), too?

[jri]

Once DM is started the gogo shell *is* your command line. It occupies the terminal from where you've started.

What do you want to do? Running the DM server as a daemon?

[JuergeN]

Exactly! (Thanks! :)) Yes, I want to run it as a daemon and write a start/stop script.

[jri]

I understand (:-)

Frankly spoken, I don't know how to stop Felix when running as a daemon.
There are some discussions on the web about it, but no apparent way to go.

As a quick hack you could simply ommit a proper stop script.
(Maybe Felix shuts down properly when receiving a kill signal. But I don't know and I'm not knowing much about start/stop scripts either, sorry)

Investigation is required here.

[jri]

One hint: I do run a DM instance at our server myself. It is put in the background and detached from the terminal via screen.
You could try this in the start script:

screen -d -m /path/to/deepamehta/deepamehta-linux.sh

Still, I don't know about stopping. As a quick hack you could try to ommit a proper stop script. In case of Neo4j is not shutdown cleanly, it performs a recovery at next start time. I didn't experience any data loss so far.

[JuergeN]

interesting: ​http://www.ralfebert.de/blog/osgi/debian_initd_daemon/

[DanielP]

1) i tried with start/stop deamon as stated in comment:8. Java process is starting, but shortly afterwards terminating itself. Seems, that calling the browser URL feature can cause the trouble.

2) tried to put it into apache karaf, felix beeing the base of it but featuring a lot of admin tools (start /stop, management-console over ssh, rmi, jmx, etc). Its all starting - but could not figure out the url to call.

next steps would be to debug why its crashing when started as daemon (logging debug to file ...)

[jri]

Seems, that calling the browser URL feature can cause the trouble.

Most likely the browser launch feature is not a problem. A possible (Desktop API) exception is catched and logged and control flow continues. It definitely works in headless environments and when running in background (see comment:7)

One issue might be standard input. DM comes with the interactive Gogo shell that waits for user input. Possibly this disrupts the background process.

... but could not figure out the url to call

The DM webclient URL is ​http://localhost:8080/

Welcome Daniel!

[JuergeN]

Today I have installed DeepaMehta ( BuildFromSource ) on my webserver. There is of course no browser installed. The jetty/felix system starts in headless mode and here we are!

Now I can start and stop DeepaMehta with the init script as described.

So, I think it would be good to add a command line option which should allow to start the server in headless mode manually even if there is a browser installed on the system (or implement another likewise solution).

Also, it would be very could if one could configure a logfile for file based logging.

[jri]

Also, it would be very could if one could configure a logfile for file based logging.

When build from source the logging is configurable in
./deepamehta-distribution/conf/logging.properties

To log to a file instead of the console change the first line from

handlers = java.util.logging.ConsoleHandler

to

handlers = java.util.logging.FileHandler

By default the logfile is created in the users home directory and is called java<nr>.log
The filename and a lot more, e.g. the log format (default is XML) and log rotation, is configurable in logging.properties as well.
DM makes use of the standard Java Util Logging API.
For file-based logging configuration see ​http://download.oracle.com/javase/6/docs/api/java/util/logging/FileHandler.html

When using the binary distribution the config file is
./conf/logging.properties

[DanielP]

sample logging.properties for both logging on console and into file. Currently does not rotate.

handlers = java.util.logging.ConsoleHandler, java.util.logging.FileHandler

# You can set the log-level at a per-class basis.
# State the fully qualified class name and add ".level"
# Possible values are e.g.
#   WARNING - high level, sparse logging
#   INFO    - mid level, the default
#   FINE    - lower level, verbose logging
# Enabling logging at a given level also enables logging at all higher levels.
# Their is an additional level:
#   OFF     - turn off logging

# log level for console
java.util.logging.ConsoleHandler.level=WARNING

# log level for logfile
java.util.logging.FileHandler.level=ALL
java.util.logging.FileHandler.pattern=logs/deepamehta%g.log
java.util.logging.FileHandler.append=true
java.util.logging.FileHandler.formatter=java.util.logging.SimpleFormatter

de.deepamehta.core.storage.neo4j.Neo4jStorage.level = WARNING
de.deepamehta.plugins.accesscontrol.AccessControlPlugin.level = FINE
# Note: FINE level doesn't work! Messages logged via logger.fine() do not appear on the console.
# That's strange and annoying.

[JuergeN]

Mmmmh, today I tried the same thing with the binary package again, but this still does not work! The service starts and then stops again after a few seconds. This needs more debugging!

[JuergeN]

Might also be interesting: ​https://github.com/kgilmer/knapsack - but did not run out of the box either :(.

[JuergeN]

On doing further research I also found an Apache project called JSVC: ​http://commons.apache.org/daemon/jsvc.html

[JuergeN]

Shit, I forgot about that part, which I had read already some time ago:

Now, there is a way to make a Java process behave like a proper daemon, jsvc. This is for example used by Tomcat. For this to work, the software needs to implement a Daemon interface from Commons Daemon. There is such an implementation for Felix ​org.apache.felix.daemon. Unfortunately it is outdated!

[Jörg Richter]

Runtime: revise logging.properties (#46)

Add note how to switch on file logging.

See ticket 46.

• Changeset: 5d260a2243276a2ce505c2c19311f0a6b4b4e934
• URL: ​https://github.com/jri/deepamehta/commit/5d260a2243276a2ce505c2c19311f0a6b4b4e934

[JuergeN]

Ok, after another talk with Danny we decided on two strategies:

1. Hotfix the error in headless mode so that DeepaMehta can be simply daemonized with the start-stop-script.

2. For enterprize level server hosting DeepaMehta will be integrated into Apache KARAF in the future.

[DanielP]

2) +1 ack

I have it sucessfully running in karaf. the only thing missing seems to be the webclient jetty-endpoint (aka URL)

[JuergeN]

When I start DeepaMehta with my init-script and I redirecte the output of the deepamehta-linux.sh script:

exec java -Dfile.encoding=UTF-8 -Djava.awt.headless=true -Dfelix.system.properties=file:conf/config.properties -jar bin/felix.jar >>/opt/deepamehta/logs/daemon.log 2>&1

here is what I get:

____________________________
Welcome to Apache Felix Gogo

g! gosh: stopping framework
[INFO] Started jetty 6.1.x at port(s) HTTP:8180
29-Aug-2011 16:02:37 de.deepamehta.plugins.webclient.WebclientPlugin allPluginsReadyHook
WARNING: ### Launching webclient failed (java.awt.HeadlessException: 
No X11 DISPLAY variable was set, but this program performed an operation which requires it.)
ERROR: Bundle org.neo4j.kernel [18] Error stopping bundle. (java.lang.IllegalArgumentException: Listener is null)
java.lang.IllegalArgumentException: Listener is null
	at org.apache.felix.framework.util.EventDispatcher.removeListener(EventDispatcher.java:298)
	at org.apache.felix.framework.Felix.removeBundleListener(Felix.java:2750)
	at org.apache.felix.framework.BundleContextImpl.removeBundleListener(BundleContextImpl.java:222)
	at org.neo4j.kernel.impl.osgi.OSGiActivator.stop(OSGiActivator.java:64)
	at org.apache.felix.framework.util.SecureAction.stopActivator(SecureAction.java:651)
	at org.apache.felix.framework.Felix.stopBundle(Felix.java:2232)
	at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1208)
	at org.apache.felix.framework.StartLevelImpl.run(StartLevelImpl.java:264)
	at java.lang.Thread.run(Thread.java:636)
ERROR: Bundle org.neo4j.kernel [18] Error stopping file:/opt/deepamehta/bundle/neo4j-kernel-1.2-1.2.jar (org.osgi.framework.BundleException: Activator stop error in bundle org.neo4j.kernel [18].)
java.lang.IllegalArgumentException: Listener is null
	at org.apache.felix.framework.util.EventDispatcher.removeListener(EventDispatcher.java:298)
	at org.apache.felix.framework.Felix.removeBundleListener(Felix.java:2750)
	at org.apache.felix.framework.BundleContextImpl.removeBundleListener(BundleContextImpl.java:222)
	at org.neo4j.kernel.impl.osgi.OSGiActivator.stop(OSGiActivator.java:64)
	at org.apache.felix.framework.util.SecureAction.stopActivator(SecureAction.java:651)
	at org.apache.felix.framework.Felix.stopBundle(Felix.java:2232)
	at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1208)
	at org.apache.felix.framework.StartLevelImpl.run(StartLevelImpl.java:264)
	at java.lang.Thread.run(Thread.java:636)

So hopefully this can help you debug and fix this error. As mentioned before I think it would be good to switch to headless mode by a command line option. (This could e.g. also enable logging to file automaticly.)

Thx!!!

Juergen

[JuergeN]

Replying to DanielP:

2) +1 ack

I have it sucessfully running in karaf. the only thing missing seems to be the webclient jetty-endpoint (aka URL)

I have opened a new ticket #87 for this purpose.

[jri]

WARNING: ### Launching webclient failed (java.awt.HeadlessException? ...

This warning is not a problem. In fact DM catches that HeadlessException? and issues that very warning. Afterwards the regular control flow is resumed.

An extra command line option to suppress the webclient start would make no difference to the daemon problem (a command line option might be a good idea in other regards anyway :-)

ERROR: Bundle org.neo4j.kernel [18] Error stopping bundle ...

This error is not a problem. This is a known bug in Neo4j 1.2 that occurs ALWAYS when stopping the Neo4j bundle.

The original problem is that one that CAUSES Felix to shutdown (the Neo4j error is just a RESULT of the Felix shutdown).

The original problem remains unknown for the moment. More detective work is required.

[jri]

As far as I know the state of affairs: the daemon start problem problem occurs only with the binary distribution and only when started in background. This supports my original hypothesis:

One issue might be standard input. DM comes with the interactive Gogo shell that waits for user input. Possibly this disrupts the background process.

(That the development version (build from source) shows no problem might be related to its different thread/process handling in conjunction with its hot-deployment feature).

Please test the binary distribution without the Gogo shell:
1) Move the 3 Gogo bundles out of the bundle/ directory
2) Put this file instead into the bundle/ directory:

​http://www.apache.org/dist/felix/org.apache.felix.log-1.0.1.jar

3) Remove the bundle-cache/ directory
4) Start DM as daemon in the background

Please report any exceptions (look at both, stdout and stderr!)

[dgf]

after some testing and study of the start-stop-daemon command, i realized that the background forking is problematic

the following exception is thrown immediately at jetty startup

INFO: Adding Event Admin service to plugin "DeepaMehta 4 Contacts"
____________________________
Welcome to Apache Felix Gogo

g! gosh: stopping framework
[WARNING] unavailable
java.lang.IllegalStateException: Can only add listeners while bundle is active or activating.
	at org.apache.felix.framework.Felix.addServiceListener(Felix.java:2772)
	at org.apache.felix.framework.BundleContextImpl.addServiceListener(BundleContextImpl.java:246)
	at org.osgi.util.tracker.ServiceTracker.open(ServiceTracker.java:308)
	at org.osgi.util.tracker.ServiceTracker.open(ServiceTracker.java:273)
	at org.apache.felix.http.base.internal.HttpServiceController.register(HttpServiceController.java:126)
	at org.apache.felix.http.base.internal.DispatcherServlet.init(DispatcherServlet.java:48)

for further explorations i installed tomcat and jetty, both java servlet containers come with a specific init.d script that sets additional java VM options etc.

in my opinion we should develop a similar script, the jetty one looks good as a starting point

[JuergeN]

Replying to jri:

Please test the binary distribution without the Gogo shell:
1) Move the 3 Gogo bundles out of the bundle/ directory
2) Put this file instead into the bundle/ directory:

​http://www.apache.org/dist/felix/org.apache.felix.log-1.0.1.jar

3) Remove the bundle-cache/ directory
4) Start DM as daemon in the background

Please report any exceptions (look at both, stdout and stderr!)

Nothing left to report! ;-) This way I can start and stop deepamehta with my script and run it in the background without any trouble. So what does this mean for the satnadard distribution?

[jri]

Replying to dgf:

java.lang.IllegalStateException?: Can only add listeners while bundle is active or activating.

I encounter that very error once in a while when refreshing the jetty bundle (not in daemon mode). For me it is not reproducable.

This error is most likely not related to the original daemon/background problem. It needs to be fixed anyway.

[jri]

Replying to JuergeN:

Nothing left to report! ;-) This way I can start and stop deepamehta with my script and run it in the background without any trouble. So what does this mean for the satnadard distribution?

Great to hear!

So, the problem is tracked down now!
(Apparently it is reading from stdin from background processes.)

For the DM standard distribution we could simply remove the gogo bundles and add the log bundle. Actually the Gogo shell is not strictly required. It is mainly used for stopping DM. Without Gogo shell the standalone user would stop DM by ctrl-C and in daemon mode the DM-process could be killed via PID. These 2 shutdown methods seems to work properly. According to the logs no DB recovery is performed while next start.

In case of shutting down this way proves problematic later on, we could consider a remote shell (via telnet):
​http://felix.apache.org/site/apache-felix-remote-shell.html

[DanielP]

a remote shell (via ssh) is available / included in apache karaf - see #87

[Jörg Richter]

Runtime: DM server runs in background (#46).

• Drop Gogo shell bundles from distribution.
• Export package org.osgi.service.log (it is needed by Jetty and was formerly exported by Gogo).

End users: stop DM by Ctrl-C.
Server: stop DM by killing via PID.

See ticket 46.

• Changeset: 5f2ec458732c4f7a733c894c9a3b231164f36875
• URL: ​https://github.com/jri/deepamehta/commit/5f2ec458732c4f7a733c894c9a3b231164f36875

[Malte]

Replying to JuergeN:

This way I can start and stop deepamehta with my script and run it in the background without any
trouble. So what does this mean for the standard distribution?

I think that this change ("dropped gogo shell" - https://trac.deepamehta.de/changeset/5f2ec458732c4f7a733c894c9a3b231164f36875/deepamehta) means for the standard distribution that we are even more in the need to give our users a new mechanism to install, activate, deinstall bundles/deepamehta-plugins for an installation..

maybe there is an existing workaround i am currently not aware of, it would be great to hear about that.. in any case i started a new ticket for discussing on how we want to be able to use a deepamehta "plugin-manager".. #96

[jri]

The background problem is solved by removing the Gogo shell from the binary distribution.

[jri]

Sorry, Malte, I missed your posting.

Replying to Malte:

I think that this change ("dropped gogo shell" - means for the standard distribution that we are even more in the need to give our users a new mechanism to install, activate, deinstall bundles/deepamehta-plugins for an installation..

With the binary distribution you can install extra bundles by putting them in the "bundle" directory and restarting.

[Malte]

thanks for the clarification! i found a new "bundle" and "bundle-cache" directory in the home-directory of the binary disribution and this mechanism descibed here works with the latest 2 snapshots, just copy the "plugin.jar"-file in the "bundle" directory and start/restart deepamehta.

[jri]

Milestone Build .deb Package deleted