Ticket #46 (closed Task: fixed)
start/stop DeepaMehta from command line
Reported by: | JuergeN | Owned by: | DanielP |
---|---|---|---|
Priority: | Major | Milestone: | |
Component: | DeepaMehta Standard Distribution | Version: | |
Keywords: | Cc: | dgf, Malte | |
Complexity: | 3 | Area: | |
Module: |
Description
Change History
comment:2 Changed 13 years ago by jri
In both cases DM is stopped by typing (in the gogo shell):
stop 0
comment:3 Changed 13 years ago by JuergeN
And is there any way to stop DeepaMehta from the command line (outside the gogo shell), too?
comment:4 Changed 13 years ago by jri
Once DM is started the gogo shell *is* your command line. It occupies the terminal from where you've started.
What do you want to do? Running the DM server as a daemon?
comment:5 Changed 13 years ago by JuergeN
Exactly! (Thanks! :)) Yes, I want to run it as a daemon and write a start/stop script.
comment:6 Changed 13 years ago by jri
I understand (:-)
Frankly spoken, I don't know how to stop Felix when running as a daemon.
There are some discussions on the web about it, but no apparent way to go.
As a quick hack you could simply ommit a proper stop script.
(Maybe Felix shuts down properly when receiving a kill signal. But I don't know and I'm not knowing much about start/stop scripts either, sorry)
Investigation is required here.
comment:7 Changed 13 years ago by jri
One hint: I do run a DM instance at our server myself. It is put in the background and detached from the terminal via screen.
You could try this in the start script:
screen -d -m /path/to/deepamehta/deepamehta-linux.sh
Still, I don't know about stopping. As a quick hack you could try to ommit a proper stop script. In case of Neo4j is not shutdown cleanly, it performs a recovery at next start time. I didn't experience any data loss so far.
comment:9 Changed 13 years ago by DanielP
- Owner set to DanielP
- Status changed from new to accepted
1) i tried with start/stop deamon as stated in comment:8. Java process is starting, but shortly afterwards terminating itself. Seems, that calling the browser URL feature can cause the trouble.
2) tried to put it into apache karaf, felix beeing the base of it but featuring a lot of admin tools (start /stop, management-console over ssh, rmi, jmx, etc). Its all starting - but could not figure out the url to call.
next steps would be to debug why its crashing when started as daemon (logging debug to file ...)
comment:10 Changed 13 years ago by jri
- Cc dgf added
Seems, that calling the browser URL feature can cause the trouble.
Most likely the browser launch feature is not a problem. A possible (Desktop API) exception is catched and logged and control flow continues. It definitely works in headless environments and when running in background (see comment:7)
One issue might be standard input. DM comes with the interactive Gogo shell that waits for user input. Possibly this disrupts the background process.
... but could not figure out the url to call
The DM webclient URL is http://localhost:8080/
Welcome Daniel!
comment:11 Changed 13 years ago by JuergeN
Today I have installed DeepaMehta ( BuildFromSource ) on my webserver. There is of course no browser installed. The jetty/felix system starts in headless mode and here we are!
Now I can start and stop DeepaMehta with the init script as described.
So, I think it would be good to add a command line option which should allow to start the server in headless mode manually even if there is a browser installed on the system (or implement another likewise solution).
Also, it would be very could if one could configure a logfile for file based logging.
comment:12 Changed 13 years ago by jri
Also, it would be very could if one could configure a logfile for file based logging.
When build from source the logging is configurable in
./deepamehta-distribution/conf/logging.properties
To log to a file instead of the console change the first line from
handlers = java.util.logging.ConsoleHandler
to
handlers = java.util.logging.FileHandler
By default the logfile is created in the users home directory and is called java<nr>.log
The filename and a lot more, e.g. the log format (default is XML) and log rotation, is configurable in logging.properties as well.
DM makes use of the standard Java Util Logging API.
For file-based logging configuration see http://download.oracle.com/javase/6/docs/api/java/util/logging/FileHandler.html
When using the binary distribution the config file is
./conf/logging.properties
comment:13 Changed 13 years ago by DanielP
sample logging.properties for both logging on console and into file. Currently does not rotate.
handlers = java.util.logging.ConsoleHandler, java.util.logging.FileHandler # You can set the log-level at a per-class basis. # State the fully qualified class name and add ".level" # Possible values are e.g. # WARNING - high level, sparse logging # INFO - mid level, the default # FINE - lower level, verbose logging # Enabling logging at a given level also enables logging at all higher levels. # Their is an additional level: # OFF - turn off logging # log level for console java.util.logging.ConsoleHandler.level=WARNING # log level for logfile java.util.logging.FileHandler.level=ALL java.util.logging.FileHandler.pattern=logs/deepamehta%g.log java.util.logging.FileHandler.append=true java.util.logging.FileHandler.formatter=java.util.logging.SimpleFormatter de.deepamehta.core.storage.neo4j.Neo4jStorage.level = WARNING de.deepamehta.plugins.accesscontrol.AccessControlPlugin.level = FINE # Note: FINE level doesn't work! Messages logged via logger.fine() do not appear on the console. # That's strange and annoying.
comment:14 Changed 13 years ago by JuergeN
Mmmmh, today I tried the same thing with the binary package again, but this still does not work! The service starts and then stops again after a few seconds. This needs more debugging!
comment:15 Changed 13 years ago by JuergeN
Might also be interesting: https://github.com/kgilmer/knapsack - but did not run out of the box either :(.
comment:16 follow-up: ↓ 17 Changed 13 years ago by JuergeN
On doing further research I also found an Apache project called JSVC: http://commons.apache.org/daemon/jsvc.html
comment:17 in reply to: ↑ 16 Changed 13 years ago by JuergeN
Shit, I forgot about that part, which I had read already some time ago:
Now, there is a way to make a Java process behave like a proper daemon, jsvc. This is for example used by Tomcat. For this to work, the software needs to implement a Daemon interface from Commons Daemon. There is such an implementation for Felix org.apache.felix.daemon. Unfortunately it is outdated!
comment:18 Changed 13 years ago by Jörg Richter
comment:19 Changed 13 years ago by JuergeN
Ok, after another talk with Danny we decided on two strategies:
- Hotfix the error in headless mode so that DeepaMehta can be simply daemonized with the start-stop-script.
- For enterprize level server hosting DeepaMehta will be integrated into Apache KARAF in the future.
comment:20 follow-up: ↓ 22 Changed 13 years ago by DanielP
2) +1 ack
I have it sucessfully running in karaf. the only thing missing seems to be the webclient jetty-endpoint (aka URL)
comment:21 Changed 13 years ago by JuergeN
When I start DeepaMehta with my init-script and I redirect the output of the deepamehta-linux.sh script:
exec java -Dfile.encoding=UTF-8 -Djava.awt.headless=true -Dfelix.system.properties=file:conf/config.properties -jar bin/felix.jar >>/opt/deepamehta/logs/daemon.log 2>&1
here is what I get:
____________________________ Welcome to Apache Felix Gogo g! gosh: stopping framework [INFO] Started jetty 6.1.x at port(s) HTTP:8180 29-Aug-2011 16:02:37 de.deepamehta.plugins.webclient.WebclientPlugin allPluginsReadyHook WARNING: ### Launching webclient failed (java.awt.HeadlessException: No X11 DISPLAY variable was set, but this program performed an operation which requires it.) ERROR: Bundle org.neo4j.kernel [18] Error stopping bundle. (java.lang.IllegalArgumentException: Listener is null) java.lang.IllegalArgumentException: Listener is null at org.apache.felix.framework.util.EventDispatcher.removeListener(EventDispatcher.java:298) at org.apache.felix.framework.Felix.removeBundleListener(Felix.java:2750) at org.apache.felix.framework.BundleContextImpl.removeBundleListener(BundleContextImpl.java:222) at org.neo4j.kernel.impl.osgi.OSGiActivator.stop(OSGiActivator.java:64) at org.apache.felix.framework.util.SecureAction.stopActivator(SecureAction.java:651) at org.apache.felix.framework.Felix.stopBundle(Felix.java:2232) at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1208) at org.apache.felix.framework.StartLevelImpl.run(StartLevelImpl.java:264) at java.lang.Thread.run(Thread.java:636) ERROR: Bundle org.neo4j.kernel [18] Error stopping file:/opt/deepamehta/bundle/neo4j-kernel-1.2-1.2.jar (org.osgi.framework.BundleException: Activator stop error in bundle org.neo4j.kernel [18].) java.lang.IllegalArgumentException: Listener is null at org.apache.felix.framework.util.EventDispatcher.removeListener(EventDispatcher.java:298) at org.apache.felix.framework.Felix.removeBundleListener(Felix.java:2750) at org.apache.felix.framework.BundleContextImpl.removeBundleListener(BundleContextImpl.java:222) at org.neo4j.kernel.impl.osgi.OSGiActivator.stop(OSGiActivator.java:64) at org.apache.felix.framework.util.SecureAction.stopActivator(SecureAction.java:651) at org.apache.felix.framework.Felix.stopBundle(Felix.java:2232) at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1208) at org.apache.felix.framework.StartLevelImpl.run(StartLevelImpl.java:264) at java.lang.Thread.run(Thread.java:636)
So hopefully this can help you debug and fix this error. As mentioned before I think it would be good to switch to headless mode by a command line option. (This could e.g. also enable logging to file automaticly.)
Thx!!!
Juergen
comment:22 in reply to: ↑ 20 Changed 13 years ago by JuergeN
comment:23 Changed 13 years ago by jri
WARNING: ### Launching webclient failed (java.awt.HeadlessException? ...
This warning is not a problem. In fact DM catches that HeadlessException? and issues that very warning. Afterwards the regular control flow is resumed.
An extra command line option to suppress the webclient start would make no difference to the daemon problem (a command line option might be a good idea in other regards anyway :-)
ERROR: Bundle org.neo4j.kernel [18] Error stopping bundle ...
This error is not a problem. This is a known bug in Neo4j 1.2 that occurs ALWAYS when stopping the Neo4j bundle.
The original problem is that one that CAUSES Felix to shutdown (the Neo4j error is just a RESULT of the Felix shutdown).
The original problem remains unknown for the moment. More detective work is required.
comment:24 follow-up: ↓ 26 Changed 13 years ago by jri
As far as I know the state of affairs: the daemon start problem problem occurs only with the binary distribution and only when started in background. This supports my original hypothesis:
One issue might be standard input. DM comes with the interactive Gogo shell that waits for user input. Possibly this disrupts the background process.
(That the development version (build from source) shows no problem might be related to its different thread/process handling in conjunction with its hot-deployment feature).
Please test the binary distribution without the Gogo shell:
1) Move the 3 Gogo bundles out of the bundle/ directory
2) Put this file instead into the bundle/ directory:
http://www.apache.org/dist/felix/org.apache.felix.log-1.0.1.jar
3) Remove the bundle-cache/ directory
4) Start DM as daemon in the background
Please report any exceptions (look at both, stdout and stderr!)
comment:25 follow-up: ↓ 27 Changed 13 years ago by dgf
after some testing and study of the start-stop-daemon command, i realized that the background forking is problematic
the following exception is thrown immediately at jetty startup
INFO: Adding Event Admin service to plugin "DeepaMehta 4 Contacts" ____________________________ Welcome to Apache Felix Gogo g! gosh: stopping framework [WARNING] unavailable java.lang.IllegalStateException: Can only add listeners while bundle is active or activating. at org.apache.felix.framework.Felix.addServiceListener(Felix.java:2772) at org.apache.felix.framework.BundleContextImpl.addServiceListener(BundleContextImpl.java:246) at org.osgi.util.tracker.ServiceTracker.open(ServiceTracker.java:308) at org.osgi.util.tracker.ServiceTracker.open(ServiceTracker.java:273) at org.apache.felix.http.base.internal.HttpServiceController.register(HttpServiceController.java:126) at org.apache.felix.http.base.internal.DispatcherServlet.init(DispatcherServlet.java:48)
for further explorations i installed tomcat and jetty, both java servlet containers come with a specific init.d script that sets additional java VM options etc.
in my opinion we should develop a similar script, the jetty one looks good as a starting point
comment:26 in reply to: ↑ 24 ; follow-ups: ↓ 28 ↓ 31 Changed 13 years ago by JuergeN
Replying to jri:
Please test the binary distribution without the Gogo shell:
1) Move the 3 Gogo bundles out of the bundle/ directory
2) Put this file instead into the bundle/ directory:
http://www.apache.org/dist/felix/org.apache.felix.log-1.0.1.jar
3) Remove the bundle-cache/ directory
4) Start DM as daemon in the background
Please report any exceptions (look at both, stdout and stderr!)
Nothing left to report! ;-) This way I can start and stop deepamehta with my script and run it in the background without any trouble. So what does this mean for the satnadard distribution?
comment:27 in reply to: ↑ 25 Changed 13 years ago by jri
Replying to dgf:
java.lang.IllegalStateException?: Can only add listeners while bundle is active or activating.
I encounter that very error once in a while when refreshing the jetty bundle (not in daemon mode). For me it is not reproducable.
This error is most likely not related to the original daemon/background problem. It needs to be fixed anyway.
comment:28 in reply to: ↑ 26 Changed 13 years ago by jri
Replying to JuergeN:
Nothing left to report! ;-) This way I can start and stop deepamehta with my script and run it in the background without any trouble. So what does this mean for the satnadard distribution?
Great to hear!
So, the problem is tracked down now!
(Apparently it is reading from stdin from background processes.)
For the DM standard distribution we could simply remove the gogo bundles and add the log bundle. Actually the Gogo shell is not strictly required. It is mainly used for stopping DM. Without Gogo shell the standalone user would stop DM by ctrl-C and in daemon mode the DM-process could be killed via PID. These 2 shutdown methods seems to work properly. According to the logs no DB recovery is performed while next start.
In case of shutting down this way proves problematic later on, we could consider a remote shell (via telnet):
http://felix.apache.org/site/apache-felix-remote-shell.html
comment:29 Changed 13 years ago by DanielP
a remote shell (via ssh) is available / included in apache karaf - see #87
comment:30 Changed 13 years ago by Jörg Richter
Runtime: DM server runs in background (#46).
- Drop Gogo shell bundles from distribution.
- Export package org.osgi.service.log (it is needed by Jetty and was formerly exported by Gogo).
End users: stop DM by Ctrl-C.
Server: stop DM by killing via PID.
See ticket 46.
comment:31 in reply to: ↑ 26 ; follow-up: ↓ 33 Changed 13 years ago by Malte
Replying to JuergeN:
This way I can start and stop deepamehta with my script and run it in the background without any
trouble. So what does this mean for the standard distribution?
I think that this change ("dropped gogo shell" - https://trac.deepamehta.de/changeset/5f2ec458732c4f7a733c894c9a3b231164f36875/deepamehta) means for the standard distribution that we are even more in the need to give our users a new mechanism to install, activate, deinstall bundles/deepamehta-plugins for an installation..
maybe there is an existing workaround i am currently not aware of, it would be great to hear about that.. in any case i started a new ticket for discussing on how we want to be able to use a deepamehta "plugin-manager".. #96
comment:32 Changed 13 years ago by jri
- Status changed from accepted to closed
- Resolution set to fixed
The background problem is solved by removing the Gogo shell from the binary distribution.
comment:33 in reply to: ↑ 31 Changed 13 years ago by jri
- Cc Malte added
Sorry, Malte, I missed your posting.
Replying to Malte:
I think that this change ("dropped gogo shell" - means for the standard distribution that we are even more in the need to give our users a new mechanism to install, activate, deinstall bundles/deepamehta-plugins for an installation..
With the binary distribution you can install extra bundles by putting them in the "bundle" directory and restarting.
comment:34 Changed 13 years ago by Malte
thanks for the clarification! i found a new "bundle" and "bundle-cache" directory in the home-directory of the binary disribution and this mechanism descibed here works with the latest 2 snapshots, just copy the "plugin.jar"-file in the "bundle" directory and start/restart deepamehta.
comment:35 Changed 10 years ago by jri
- Milestone Build .deb Package deleted
Milestone Build .deb Package deleted
You can start DM from the command line.
With the binary distribution run:
(or the respective script for your platform)
When you've build from source run:
Does it work for you?