Ticket #46 (closed Task: fixed)

Opened 13 years ago

Last modified 11 years ago

start/stop DeepaMehta from command line

Reported by: JuergeN Owned by: DanielP
Priority: Major Milestone:
Component: DeepaMehta Standard Distribution Version:
Keywords: Cc: dgf, Malte
Complexity: 3 Area:
Module:

Description


Change History

comment:1 Changed 13 years ago by jri

You can start DM from the command line.

With the binary distribution run:

./deepamehta-linux.sh

(or the respective script for your platform)

When you've build from source run:

cd deepamehta
mvn pax:run

Does it work for you?

comment:2 Changed 13 years ago by jri

In both cases DM is stopped by typing (in the gogo shell):

stop 0

comment:3 Changed 13 years ago by JuergeN

And is there any way to stop DeepaMehta from the command line (outside the gogo shell), too?

comment:4 Changed 13 years ago by jri

Once DM is started the gogo shell *is* your command line. It occupies the terminal from where you've started.

What do you want to do? Running the DM server as a daemon?

comment:5 Changed 13 years ago by JuergeN

Exactly! (Thanks! :)) Yes, I want to run it as a daemon and write a start/stop script.

comment:6 Changed 13 years ago by jri

I understand (:-)

Frankly spoken, I don't know how to stop Felix when running as a daemon.
There are some discussions on the web about it, but no apparent way to go.

As a quick hack you could simply ommit a proper stop script.
(Maybe Felix shuts down properly when receiving a kill signal. But I don't know and I'm not knowing much about start/stop scripts either, sorry)

Investigation is required here.

comment:7 Changed 13 years ago by jri

One hint: I do run a DM instance at our server myself. It is put in the background and detached from the terminal via screen.
You could try this in the start script:

screen -d -m /path/to/deepamehta/deepamehta-linux.sh

Still, I don't know about stopping. As a quick hack you could try to ommit a proper stop script. In case of Neo4j is not shutdown cleanly, it performs a recovery at next start time. I didn't experience any data loss so far.

comment:9 Changed 13 years ago by DanielP

  • Owner set to DanielP
  • Status changed from new to accepted

1) i tried with start/stop deamon as stated in comment:8. Java process is starting, but shortly afterwards terminating itself. Seems, that calling the browser URL feature can cause the trouble.

2) tried to put it into apache karaf, felix beeing the base of it but featuring a lot of admin tools (start /stop, management-console over ssh, rmi, jmx, etc). Its all starting - but could not figure out the url to call.

next steps would be to debug why its crashing when started as daemon (logging debug to file ...)

comment:10 Changed 13 years ago by jri

  • Cc dgf added

Seems, that calling the browser URL feature can cause the trouble.

Most likely the browser launch feature is not a problem. A possible (Desktop API) exception is catched and logged and control flow continues. It definitely works in headless environments and when running in background (see comment:7)

One issue might be standard input. DM comes with the interactive Gogo shell that waits for user input. Possibly this disrupts the background process.

... but could not figure out the url to call

The DM webclient URL is http://localhost:8080/

Welcome Daniel!

comment:11 Changed 13 years ago by JuergeN

Today I have installed DeepaMehta ( BuildFromSource ) on my webserver. There is of course no browser installed. The jetty/felix system starts in headless mode and here we are!

Now I can start and stop DeepaMehta with the init script as described.

So, I think it would be good to add a command line option which should allow to start the server in headless mode manually even if there is a browser installed on the system (or implement another likewise solution).

Also, it would be very could if one could configure a logfile for file based logging.

comment:12 Changed 13 years ago by jri

Also, it would be very could if one could configure a logfile for file based logging.

When build from source the logging is configurable in
./deepamehta-distribution/conf/logging.properties

To log to a file instead of the console change the first line from

handlers = java.util.logging.ConsoleHandler

to

handlers = java.util.logging.FileHandler

By default the logfile is created in the users home directory and is called java<nr>.log
The filename and a lot more, e.g. the log format (default is XML) and log rotation, is configurable in logging.properties as well.
DM makes use of the standard Java Util Logging API.
For file-based logging configuration see http://download.oracle.com/javase/6/docs/api/java/util/logging/FileHandler.html

When using the binary distribution the config file is
./conf/logging.properties

Last edited 13 years ago by jri (previous) (diff)

comment:13 Changed 13 years ago by DanielP

sample logging.properties for both logging on console and into file. Currently does not rotate.

handlers = java.util.logging.ConsoleHandler, java.util.logging.FileHandler

# You can set the log-level at a per-class basis.
# State the fully qualified class name and add ".level"
# Possible values are e.g.
#   WARNING - high level, sparse logging
#   INFO    - mid level, the default
#   FINE    - lower level, verbose logging
# Enabling logging at a given level also enables logging at all higher levels.
# Their is an additional level:
#   OFF     - turn off logging

# log level for console
java.util.logging.ConsoleHandler.level=WARNING

# log level for logfile
java.util.logging.FileHandler.level=ALL
java.util.logging.FileHandler.pattern=logs/deepamehta%g.log
java.util.logging.FileHandler.append=true
java.util.logging.FileHandler.formatter=java.util.logging.SimpleFormatter

de.deepamehta.core.storage.neo4j.Neo4jStorage.level = WARNING
de.deepamehta.plugins.accesscontrol.AccessControlPlugin.level = FINE
# Note: FINE level doesn't work! Messages logged via logger.fine() do not appear on the console.
# That's strange and annoying.

comment:14 Changed 13 years ago by JuergeN

Mmmmh, today I tried the same thing with the binary package again, but this still does not work! The service starts and then stops again after a few seconds. This needs more debugging!

comment:15 Changed 13 years ago by JuergeN

Might also be interesting: https://github.com/kgilmer/knapsack - but did not run out of the box either :(.

comment:16 follow-up: ↓ 17 Changed 13 years ago by JuergeN

On doing further research I also found an Apache project called JSVC: http://commons.apache.org/daemon/jsvc.html

comment:17 in reply to: ↑ 16 Changed 13 years ago by JuergeN

Shit, I forgot about that part, which I had read already some time ago:

Now, there is a way to make a Java process behave like a proper daemon, jsvc. This is for example used by Tomcat. For this to work, the software needs to implement a Daemon interface from Commons Daemon. There is such an implementation for Felix org.apache.felix.daemon. Unfortunately it is outdated!

comment:18 Changed 13 years ago by Jörg Richter

Runtime: revise logging.properties (#46)

Add note how to switch on file logging.

See ticket 46.

comment:19 Changed 13 years ago by JuergeN

Ok, after another talk with Danny we decided on two strategies:

  1. Hotfix the error in headless mode so that DeepaMehta can be simply daemonized with the start-stop-script.
  1. For enterprize lever server hosting DeepaMehta will be integrated into Apache KARAF in the future.
Version 0, edited 13 years ago by JuergeN (next)

comment:20 follow-up: ↓ 22 Changed 13 years ago by DanielP

2) +1 ack

I have it sucessfully running in karaf. the only thing missing seems to be the webclient jetty-endpoint (aka URL)

comment:21 Changed 13 years ago by JuergeN

When I start DeepaMehta with my init-script and I redirect the output of the deepamehta-linux.sh script:

exec java -Dfile.encoding=UTF-8 -Djava.awt.headless=true -Dfelix.system.properties=file:conf/config.properties -jar bin/felix.jar >>/opt/deepamehta/logs/daemon.log 2>&1

here is what I get:

____________________________
Welcome to Apache Felix Gogo

g! gosh: stopping framework
[INFO] Started jetty 6.1.x at port(s) HTTP:8180
29-Aug-2011 16:02:37 de.deepamehta.plugins.webclient.WebclientPlugin allPluginsReadyHook
WARNING: ### Launching webclient failed (java.awt.HeadlessException: 
No X11 DISPLAY variable was set, but this program performed an operation which requires it.)
ERROR: Bundle org.neo4j.kernel [18] Error stopping bundle. (java.lang.IllegalArgumentException: Listener is null)
java.lang.IllegalArgumentException: Listener is null
	at org.apache.felix.framework.util.EventDispatcher.removeListener(EventDispatcher.java:298)
	at org.apache.felix.framework.Felix.removeBundleListener(Felix.java:2750)
	at org.apache.felix.framework.BundleContextImpl.removeBundleListener(BundleContextImpl.java:222)
	at org.neo4j.kernel.impl.osgi.OSGiActivator.stop(OSGiActivator.java:64)
	at org.apache.felix.framework.util.SecureAction.stopActivator(SecureAction.java:651)
	at org.apache.felix.framework.Felix.stopBundle(Felix.java:2232)
	at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1208)
	at org.apache.felix.framework.StartLevelImpl.run(StartLevelImpl.java:264)
	at java.lang.Thread.run(Thread.java:636)
ERROR: Bundle org.neo4j.kernel [18] Error stopping file:/opt/deepamehta/bundle/neo4j-kernel-1.2-1.2.jar (org.osgi.framework.BundleException: Activator stop error in bundle org.neo4j.kernel [18].)
java.lang.IllegalArgumentException: Listener is null
	at org.apache.felix.framework.util.EventDispatcher.removeListener(EventDispatcher.java:298)
	at org.apache.felix.framework.Felix.removeBundleListener(Felix.java:2750)
	at org.apache.felix.framework.BundleContextImpl.removeBundleListener(BundleContextImpl.java:222)
	at org.neo4j.kernel.impl.osgi.OSGiActivator.stop(OSGiActivator.java:64)
	at org.apache.felix.framework.util.SecureAction.stopActivator(SecureAction.java:651)
	at org.apache.felix.framework.Felix.stopBundle(Felix.java:2232)
	at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1208)
	at org.apache.felix.framework.StartLevelImpl.run(StartLevelImpl.java:264)
	at java.lang.Thread.run(Thread.java:636)

So hopefully this can help you debug and fix this error. As mentioned before I think it would be good to switch to headless mode by a command line option. (This could e.g. also enable logging to file automaticly.)

Thx!!!

Juergen

Last edited 13 years ago by JuergeN (previous) (diff)

comment:22 in reply to: ↑ 20 Changed 13 years ago by JuergeN

Replying to DanielP:

2) +1 ack

I have it sucessfully running in karaf. the only thing missing seems to be the webclient jetty-endpoint (aka URL)

I have opened a new ticket #87 for this purpose.

comment:23 Changed 13 years ago by jri

WARNING: ### Launching webclient failed (java.awt.HeadlessException? ...

This warning is not a problem. In fact DM catches that HeadlessException? and issues that very warning. Afterwards the regular control flow is resumed.

An extra command line option to suppress the webclient start would make no difference to the daemon problem (a command line option might be a good idea in other regards anyway :-)

ERROR: Bundle org.neo4j.kernel [18] Error stopping bundle ...

This error is not a problem. This is a known bug in Neo4j 1.2 that occurs ALWAYS when stopping the Neo4j bundle.

The original problem is that one that CAUSES Felix to shutdown (the Neo4j error is just a RESULT of the Felix shutdown).

The original problem remains unknown for the moment. More detective work is required.

comment:24 follow-up: ↓ 26 Changed 13 years ago by jri

As far as I know the state of affairs: the daemon start problem problem occurs only with the binary distribution and only when started in background. This supports my original hypothesis:

One issue might be standard input. DM comes with the interactive Gogo shell that waits for user input. Possibly this disrupts the background process.

(That the development version (build from source) shows no problem might be related to its different thread/process handling in conjunction with its hot-deployment feature).

Please test the binary distribution without the Gogo shell:
1) Move the 3 Gogo bundles out of the bundle/ directory
2) Put this file instead into the bundle/ directory:

http://www.apache.org/dist/felix/org.apache.felix.log-1.0.1.jar

3) Remove the bundle-cache/ directory
4) Start DM as daemon in the background

Please report any exceptions (look at both, stdout and stderr!)

comment:25 follow-up: ↓ 27 Changed 13 years ago by dgf

after some testing and study of the start-stop-daemon command, i realized that the background forking is problematic

the following exception is thrown immediately at jetty startup

INFO: Adding Event Admin service to plugin "DeepaMehta 4 Contacts"
____________________________
Welcome to Apache Felix Gogo

g! gosh: stopping framework
[WARNING] unavailable
java.lang.IllegalStateException: Can only add listeners while bundle is active or activating.
	at org.apache.felix.framework.Felix.addServiceListener(Felix.java:2772)
	at org.apache.felix.framework.BundleContextImpl.addServiceListener(BundleContextImpl.java:246)
	at org.osgi.util.tracker.ServiceTracker.open(ServiceTracker.java:308)
	at org.osgi.util.tracker.ServiceTracker.open(ServiceTracker.java:273)
	at org.apache.felix.http.base.internal.HttpServiceController.register(HttpServiceController.java:126)
	at org.apache.felix.http.base.internal.DispatcherServlet.init(DispatcherServlet.java:48)

for further explorations i installed tomcat and jetty, both java servlet containers come with a specific init.d script that sets additional java VM options etc.

in my opinion we should develop a similar script, the jetty one looks good as a starting point

comment:26 in reply to: ↑ 24 ; follow-ups: ↓ 28 ↓ 31 Changed 13 years ago by JuergeN

Replying to jri:

Please test the binary distribution without the Gogo shell:
1) Move the 3 Gogo bundles out of the bundle/ directory
2) Put this file instead into the bundle/ directory:

http://www.apache.org/dist/felix/org.apache.felix.log-1.0.1.jar

3) Remove the bundle-cache/ directory
4) Start DM as daemon in the background

Please report any exceptions (look at both, stdout and stderr!)

Nothing left to report! ;-) This way I can start and stop deepamehta with my script and run it in the background without any trouble. So what does this mean for the satnadard distribution?

comment:27 in reply to: ↑ 25 Changed 13 years ago by jri

Replying to dgf:

java.lang.IllegalStateException?: Can only add listeners while bundle is active or activating.

I encounter that very error once in a while when refreshing the jetty bundle (not in daemon mode). For me it is not reproducable.

This error is most likely not related to the original daemon/background problem. It needs to be fixed anyway.

comment:28 in reply to: ↑ 26 Changed 13 years ago by jri

Replying to JuergeN:

Nothing left to report! ;-) This way I can start and stop deepamehta with my script and run it in the background without any trouble. So what does this mean for the satnadard distribution?

Great to hear!

So, the problem is tracked down now!
(Apparently it is reading from stdin from background processes.)

For the DM standard distribution we could simply remove the gogo bundles and add the log bundle. Actually the Gogo shell is not strictly required. It is mainly used for stopping DM. Without Gogo shell the standalone user would stop DM by ctrl-C and in daemon mode the DM-process could be killed via PID. These 2 shutdown methods seems to work properly. According to the logs no DB recovery is performed while next start.

In case of shutting down this way proves problematic later on, we could consider a remote shell (via telnet):
http://felix.apache.org/site/apache-felix-remote-shell.html

comment:29 Changed 13 years ago by DanielP

a remote shell (via ssh) is available / included in apache karaf - see #87

comment:30 Changed 13 years ago by Jörg Richter

Runtime: DM server runs in background (#46).

  • Drop Gogo shell bundles from distribution.
  • Export package org.osgi.service.log (it is needed by Jetty and was formerly exported by Gogo).

End users: stop DM by Ctrl-C.
Server: stop DM by killing via PID.

See ticket 46.

comment:31 in reply to: ↑ 26 ; follow-up: ↓ 33 Changed 13 years ago by Malte

Replying to JuergeN:

This way I can start and stop deepamehta with my script and run it in the background without any
trouble. So what does this mean for the standard distribution?

I think that this change ("dropped gogo shell" - https://trac.deepamehta.de/changeset/5f2ec458732c4f7a733c894c9a3b231164f36875/deepamehta) means for the standard distribution that we are even more in the need to give our users a new mechanism to install, activate, deinstall bundles/deepamehta-plugins for an installation..

maybe there is an existing workaround i am currently not aware of, it would be great to hear about that.. in any case i started a new ticket for discussing on how we want to be able to use a deepamehta "plugin-manager".. #96

comment:32 Changed 13 years ago by jri

  • Status changed from accepted to closed
  • Resolution set to fixed

The background problem is solved by removing the Gogo shell from the binary distribution.

comment:33 in reply to: ↑ 31 Changed 13 years ago by jri

  • Cc Malte added

Sorry, Malte, I missed your posting.

Replying to Malte:

I think that this change ("dropped gogo shell" - means for the standard distribution that we are even more in the need to give our users a new mechanism to install, activate, deinstall bundles/deepamehta-plugins for an installation..

With the binary distribution you can install extra bundles by putting them in the "bundle" directory and restarting.

comment:34 Changed 13 years ago by Malte

thanks for the clarification! i found a new "bundle" and "bundle-cache" directory in the home-directory of the binary disribution and this mechanism descibed here works with the latest 2 snapshots, just copy the "plugin.jar"-file in the "bundle" directory and start/restart deepamehta.

comment:35 Changed 11 years ago by jri

  • Milestone Build .deb Package deleted

Milestone Build .deb Package deleted

Note: See TracTickets for help on using tickets.