GeoNetwork From Scratch I : The Phantom Catalog

GeoNetwork never has been an easy software to work with. But specially after the 3.0 version release, many things have changed. On this series of posts we will try to help new developers start with GeoNetwork.

The source code of GeoNetwork is available on a public repository on Github. This means that you can clone, fork and propose pushes of your custom changes. If you are not familiar with repositories of code or git, you should check this quick manual.

GeoNetwork is built using maven version 3+. It is written on Java and requires version 7 or more. It works both with OpenJDK or the Oracle version.You will need git and maven installed on your local machine to work. There are several ways to install this on your local machine; for example if you have a Debian based OS (like Ubuntu), you can install them with just this command:

sudo apt-get install maven git

Make sure you installed maven version 3!!

$ mvn –version
Apache Maven 3.2.1 (ea8b2b07643dbb1b84b6d16e1f08391b666bc1e9; 2014-02-14T18:37:52+01:00)
Maven home: ….

Remember that this will also install java on your system. You can check that the version is the right one with the following command:

java -version

So, the very first step once you have your environment set up is clone the GeoNetwork repository on your local machine. That can be done on the command line using the following command inside an empty folder where the source code will be populated:

cd yourEmptyFolder
git clone https://github.com/geonetwork/core-geonetwork.git
git submodule init
git submodule update

As you can see, all the source code shown on github is also available on your local machine now.

The source code of GeoNetwork is split on several smaller maven projects. To run GeoNetwork, you have to build all of them and run the project named “web“. If you are familiar to maven, you will probably have guessed that you have to run a package install command on the root folder of GeoNetwork source code. But if you try that, maven will warn you that for building GeoNetwork you need more memory than the default memory provided to maven. This means, you will have to export the maven options to increase the memory like this:

export MAVEN_OPTS=”-Xmx512M -XX:MaxPermSize=256M”

At this point we are not interested in running the tests, so you can skip them using the parameter “-DskipTests”:

mvn package install -DskipTests

At the end of this build (which can take long, depending on your network connection, as it has many third party libraries), you will see something like this:

[INFO] ————————————————————————
[INFO] Reactor Summary:
[INFO]
[INFO] GeoNetwork opensource ……………………….. SUCCESS [ 3.111 s]
[INFO] common utils ……………………………….. SUCCESS [ 13.678 s]
[INFO] Caching xslt module …………………………. SUCCESS [ 7.607 s]
[INFO] ArcSDE module (dummy-api) ……………………. SUCCESS [ 7.860 s]
[INFO] GeoNetwork domain …………………………… SUCCESS [ 33.785 s]
[INFO] Oaipmh modules ……………………………… SUCCESS [ 0.833 s]
[INFO] GeoNetwork Events …………………………… SUCCESS [ 0.654 s]
[INFO] GeoNetwork schema plugins ……………………. SUCCESS [ 4.646 s]
[INFO] GeoNetwork schema plugins core ……………….. SUCCESS [ 5.338 s]
[INFO] GeoNetwork schema plugin for ISO19139/119 standards SUCCESS [ 8.432 s]
[INFO] GeoNetwork core …………………………….. SUCCESS [ 16.304 s]
[INFO] GeoNetwork schema plugin for Dublin Core records retrieved by CSW SUCCESS [ 5.031 s]
[INFO] GeoNetwork schema plugin for Dublin Core standard . SUCCESS [ 8.419 s]
[INFO] GeoNetwork schema plugin for ISO19110 standard …. SUCCESS [ 3.627 s]
[INFO] GeoNetwork CSW server ……………………….. SUCCESS [ 5.546 s]
[INFO] GeoNetwork harvesters ……………………….. SUCCESS [ 3.888 s]
[INFO] GeoNetwork health monitor ……………………. SUCCESS [ 2.489 s]
[INFO] GeoNetwork services …………………………. SUCCESS [ 8.597 s]
[INFO] Geonetwork Web Resources 4 Java ………………. SUCCESS [ 5.261 s]
[INFO] Cobweb Customizations ……………………….. SUCCESS [ 4.226 s]
[INFO] GeoNetwork INSPIRE Atom ……………………… SUCCESS [ 3.990 s]
[INFO] Tests for schema plugins …………………….. SUCCESS [ 2.334 s]
[INFO] GeoNetwork user interface module ……………… SUCCESS [ 35.356 s]
[INFO] JS API and Service documentation ……………… SUCCESS [ 21.203 s]
[INFO] GeoNetwork web client module …………………. SUCCESS [ 47.484 s]
[INFO] GeoNetwork Web module ……………………….. SUCCESS [ 48.490 s]
[INFO] GeoNetwork E2E Javascript Tests ………………. SUCCESS [ 1.645 s]
[INFO] ————————————————————————
[INFO] BUILD SUCCESS
[INFO] ————————————————————————
[INFO] Total time: 02:19 min (Wall Clock)
[INFO] Finished at: 2015-07-17T10:36:43+01:00
[INFO] Final Memory: 232M/441M
[INFO] ————————————————————————

This will generate a war file you can use in any Java Application Container (server) like Tomcat on web/target/geonetwork.war

Congratulations! You are ready to run GeoNetwork. To do this, just go to the web folder and run jetty in there:

cd web; mvn jetty:run

After jetty starts, you can see your running GeoNetwork by opening a browser and enter to http://localhost:8080/geonetwork

Continue in Attack of the IDEs

FOSS4G 2015 – Seoul

Last week I had the privilege to attend the main osgeo conference: the FOSS4G. This time it took place on Seoul, Korea. Exotic place I strongly recommend to visit, but better to focus on non-cultural surprises on this post.

It’s impossible to write everything on a single blog post, as it was impossible to assist to all the interesting parallel threads that run on those short five days. But let me guide you through my steps so I can share part of my experience until we get access to the full videos of that awesome week.

First there were the workshops. I have to confess something: my first options for workshops were cancelled. But it doesn’t matter, as I had problems deciding on first place. So, I started with the WS02 workshop:

Exploring the Sensor Observation Service Standard Enhanced by IstSOS Special Features

Let me summarize it: if you want to work with sensors, take a very close look at IstSOS. Combining PostGIS, Python, GDAL and Apache you get not only a complete sensor service, but a complete sensor data management system. Easy to use, easy to install, easy to everything!

IstSOS UI

Quality Assurance is integrated on IstSOS to make sure the sensor doesn’t go mad. This process are sometimes done asynchronously to conflate your own sensor data with other sources. It doesn’t just return a basic boolean, but a statistic value of confidence based on previous data.

Another interesting feature is the virtual sensors, which allows you to create your “own” “sensors” based on data from real physical sensors. Useful to conflate data and offer it on a unique endpoint as sensor.

As if this workshop wasn’t good enough, we got some real arduino and sensors to play with them.

arduino

Once we finished this amazing workshop, I went to assist Florent Gravin on the GeoNetwork 3 workshop in the afternoon:

geonetwork-workshop

WS08 Build your own data portal using GeoNetwork 3

Besides some Windows issues (surprise!), I think that the workshop was very successful. Assistants not only built their own user interface style for GeoNetwork, but asked us about advanced features and how to implement them.

geonetwork-workshop2

As Florent said: “We are the experts of GeoNetwork, so if you have any questions, it is now or never.” And we had questions asked.

Jet-lagged day finished after this workshop. So we went to the hotel to try to rest a bit more and start with renewed strength the following day, which I started with:

WS19 Beyond GeoServer Basics

If you already run a GeoServer, most of this workshop is already known to you. But if you don’t… bad you missed it! There were a lot of fancy features to play with, like regexp on parameters for sql layers (surprised that very few knew what regexp meant!).

And, of course, WPS already useful in production now. You can even use WPS inside SLD definitions!

After refreshing knowledge on GeoServer, I went for the last workshop:

WS21 OpenSource 3D GIS

Yes, you are right, 3D GIS has too much hype right now. But Oslandia did a great job on this workshop. If you don’t have experience with 3D, I strongly advise you to try to follow the workshop, posted on github.

workshop-3d

We even had some docker introduction. And the second day was finished, but we still had energy to went for the last one. Or some more beers:

last-one

I was already on the first slot of the conference, during the Opening Ceremony, when I first noticed that we talked too much about open instead of free.

openopeness

Besides that, Venkatesh made very clear that we need more Geo4All, specially on developing countries.

Very interesting the national SDI of Korea presentation about the workflow of open source. While I don’t fully agree with their vision, it is always refreshing to see new perspectives on this subject. Maybe we should make more clear that free software can have a professional customer support as good as any closed source software. Probably even better in most cases, as there are more companies that can offer it.

Now the conference divided on so many threads at the same time that it was impossible to follow them all. QGIS (new useful plugins to develop) and MapServer (now faster than ever) before lunch and the sensor and crowdsourcing session after lunch. There, between the ISA server for indoor spatial data, Apache Spark, Ontologies and CartoDB, we presented Cobweb. The day couldn’t finish without a very productive and promising BOFH session about crowdsourcing… and more!

Thursday was a hard day: jet lag started to go away, but taking #geobeers every night does not help much.

Alysa Wright started with a cool presentation about why and how did she start on the spatial world. We should never forget that, as a community, we come from very different places and perspectives. And still, we share a common view about freedom.

On the last session of the day, I learned about OpenDroneMap. I couldn’t stop asking myself why didn’t I knew about this incredible project before. Where drone software is even more awesome than the drone hardware.

Finally, Paul Ramsey talked about sustainability of free and open software and why we should care about who maintains it.

After the keynote, more social drinks with lots of fun:

Last day of conference is here. Friday tastes bitterweet. And not because of the food! María Brovelli starts with a keynote about crowdsourcing and VGI and how they can change our community. Because even if we don’t deal directly with the data, the software is also affected by all this.         The next keynote was from Marco Hugentobler, talking about the qGis community from a developer’s perspective. Because, yes, “developers are human and not everyone is aware of that”. What a surprise! In any case, never trust someone with the “expert” label, just trust the community.           After the coffee break, I attended a very instructive presentation from Henrik Lund Pedersen about caching fresh data. Is it possible to have a cached and updated server for GIS data?

During lunch we had some lightning random talks and I couldn’t stop myself. Please, use free and not open:

Jody Garnett talked about documentation and why we should be very careful when writing it: not only to help advanced developers but also to not scare newbies. Never use the word “easy”, for example. If you use it and the reader don’t find it easy, he will go away to look for something easier for him. And maybe the original documentation was easy, it’s just that it hasn’t been updated with new features.

Then we had another Cobweb session on the main grand ballroom, this time by Panos, talking about the mobile part of the framework.

And FOSS4G was finishing fast! But not before our president Jeff gave his vision about OsGeo and where we should focus on. Build community is the key. And shaking hands with people wearing suits.

On top of all the final party, GeoCat gave the first Most Innovative Developer Award. We hope to repeat it year after year encouraging developers in osgeo software.

Before finishing this post, I want to say a very big thank you to all the sponsors. Without them, this is not possible.

See you next year in Bonn!

 

Published first in GeoCat.

(English) SigLibre 9, Girona

The 9th SIGLibre conference in Girona starts with bad news: no wifi. But that didn’t stop us to talk about geo free stuff.

Almost all the plenary talks touched topis like smart citizens, open data, crowd data, crowd sourcing,… It is quite clear this is an emerging subject that we are going to use more and more. But, is this really a business market or is this just something to research and contribute to the community?

Malcolm Bain – IdLawPartners
“Open core, dual licensing, master subscription, commercial open source product specialists – mucha jerga para hablar de cómo hacer dinero con Open source: Hablemos de sus aspectos legales”

What makes a software free is not the code: it is the license. And that’s what is important. But Malcolm hasn’t come here to talk about licenses or patents, but about business with free software. We need to find the added value for open software so we are able to sell it. The problem is: this added value has nothing to do with FOSS. FOSS is freaky, geeky, sneaky,… How can we mix this two concepts? Where is the value position?

The added value for FOSS based business is the legal part. Open source license grants you access, modify and distribute the source code, so you can “physically” fix the bugs yourself, get a local IT supplier to support you (or support yourself) and share that support across a community. You can also have your own agenda on the versions you use. There is an unlimited license to use your free software on the life cycle you want, you don’t depend on third parties company’s planning.

And most important: you can check the code you are being sold. No seller can cheat you selling software that doesn’t do what it needs to do.

There are many strategies to sell free software:

* Dual License: two types of licenses for the same code: private or free.
* Open Core: free core but extensions are payed.
* Subscription strategy: you sell the technical support only.
* Product specialist: revenues from services (maintenance and consulting).
* Consulting services strategy: pure service model.
* Hosted strategy: Saas, cloud.
* Platform providers: for example, openstack hosting.
* Legal services: consulting legal services.

The free software is an investment on the long run, which is something that capitalists investors usually don’t like. That’s why free software is going to win: because there are no capitalists investors that want to get their interests back.

We live in a world that is designed, built and run by Open Source software. Don’t forget it!

Alberto Labarga – NavarraBioMed
“Periodismo de datos y la visualización de datos abiertos”

Alberto works on health and medical services based on free software and free hardware. As a sidenote: he mentioned that all gis software he knew was free, so it looks like we are doing marketing stuff right. He talked about open data and how to display it. Sometimes we focus too much on technology and not on how our users perceive it. Right now, everyone claims to be doing big data visualization, but very few people has real access to big data databases. So, there is a lot of fuzzy smoke on this topic.

From the crowd data perspective, there are many interesting projects like #adoptaUnSenador or #adoptaUnCorrupto, where spanish government obscured PDFs about incomings and wealth of politicians were crowd-parsed. There are other projects, like @jjelosua’s “España en llamas”, which conflates all data about fires in Spain. As this kind of projects appeared, many tools were also developed to help this type of jobs. Transparency comes hand by hand with this projects, like visualizing budget spent on each public administration.

The biggest problem with opendata right now is that there is a lot of public data still not published. And what is published is usually static or not on a useful format. Spain (and Europe) still have to follow UK on this topic and open more data so more apps and usecases can be developed.

He had an amazing presentation where he presented more usecases I can write on a summary here.

Irene Compte – Exploradora y aventurera tecnológica
“Open Smart cities: mitos y realidades”

Irene did an interactive theatre presentation showing his view about how this smartcity concept started and evolved. She has been working on this topic since the very beginning of the concept and that made her have a very close experience.

Victor Olaya – Boundless
“Algunas ideas sobre música, literatura y mapas”

Closing plenaries, Victor Olaya enphasizes the need to do good maps. We have a lot of technical people working on maps, building maps. But, do they have some studies about it? Are they prepared to present data the best way? We assume that all maps are used the same way all across the world. And that we can use almost the same style for all usecases. All this assumptions only complicates things to the user in the end.

Ed Parsons – Google
“Where is the Cathedral and the Bazaar : some musings on openness, ecosystems and bringing geo to the web”

At the end of the day, as a closing plenary, we had Ed Parsons, from Google, who talked about what is his view on future hits on geospatial. Appart from his insistence on using the word “open” instead of “free” or “libre” (they are difficult non-marketable words, he said), there is a very clear position on Google trying to go further on their attempt to control what people do. “Only if you want, the choice to have a mobile phone is yours”, as he clearly states. But is this true? Can someone live on our society without using mobile phones?

For example, Google is working on backpacks to map inside buildings as their streetview cars map cities. This way, they will have more detailed information about where people are and what they do. They claim it is for offering more detailed services. I don’t doubt it but, where is intimacy going? Will new generations know what is to have a private space where they don’t need to pose? Where they can relax because no-one is watching? Are we coming closer to a 1984 Big Brother based on companies instead of Government? Will that may be even worse, even if the company’s motto is “don’t be evil”?

A lot of questions to answer.

Original entry in GeoCat’s blog

Jornadas SIG Libre Girona VII

Estas últimas Jornadas SIG Libre de Girona han girado en torno a servicios cloud y datos abiertos. Desde el apoteósico inicio con las ponencias plenarias, con parte destacada de Sergi Morales (ExportosenTI), el resto de las charlas han ido rodando todas en el mismo tema. Algunas quizás un poco más críticas, como F. Puga desde CartoLab cuando nos pidió que no olvidáramos que no todo el mundo tiene acceso global a internet, y que muchos millones de personas, las cuales no tienen nuestro nivel tecnológico, también tienen necesidades GIS.

Esto es lo que significa cloud en zonas en desarrollo. #siglibre7 #firstworldproblems

Otra gran cuestión que se ha levantado en estas jornadas ha sido, de la mano de Malcolm Bain, cuales son los límites legales de los servicios de almacenamiento de datos en la nube, qué podemos esperar y exigir y hasta qué punto, desde la perspectiva del proveedor de servicios, tenemos que ofrecer un mínimo de nivel de servicio. Resulta sin duda sorprendente aprender que algunos de los contratos que aceptamos en servicios muy conocidos (como correo electrónico web o hosting) son, sencillamente, ilegales.

La batalla del software libre ya está ganada, ahora queda la batalla de la libertad y privacidad de los datos.

También ha resultado una delicia ver el enfrentamiento dialéctico entre varios de los ponentes, como cuando Javier de la Torre (Cartodb) arremetió contra OGC o las IDE. Sin duda un tema que, aunque no coincido completamente con su opinión, es algo a debatir y mejorar. ¿Deberían intentar los IDE hacerse más amigables para acercarse al usuario o es suficiente con ser un repositorio de datos?

El jueves por la tarde, Geocat tuvo sesión doble entre varias charlas muy centradas en los metadatos y su importancia en los IDE. Primero presentamos rápidamente quiénes éramos y qué hacíamos, centrándonos sobre todo en GeoNetwork y Bridge, mencionando también nuestro futuro GeoCat Live. Y justo antes de terminar el día, un taller de 30 minutos para presentar GeoNetwork, justo antes de dar paso a la primera reunión de geoinquietos nacionales.

Es difícil resumir en un sólo artículo toda la tremenda dimensión alcanzada en estas jornadas. Mucho optimismo, muchas ganas de seguir trabajando y, sobre todo, la certeza de que estamos en el camino correcto, apoyando el software y los datos libres, centrándonos en la parte social de nuestro trabajo. El viernes, casi para finalizar, Javier Sánchez orientó la recta final de las jornadas hacia este tema, hablando de las empresas sociales, las cuales no sólo tienen una cuenta de resultados económica sino también social.

En resumen, las Jornadas SIG Libre Girona son sin duda el Evento SIG(en mayúsculas) que cualquier hispano hablante debería tener en cuenta si quiere estar al día de las últimas novedades.The main theme of this Jornadas SIG Libre de Girona has been cloud services and open data. Since the apotheosic beginning, with an outstanding speech of Sergi Morales (ExportosenTI), the rest of the conference has been running around the same theme. Some of them maybe more critical, like the one of F. Puga from CartoLab when he asked us not to forget undevelopment zones where not everyone has internet access and millions  of people, which doesn’t have our tech level, also have GIS needs.

This is what cloud means on undevelopment countries. #siglibre7 #firstworldproblems

Anotheer interesting theme of this conference came by the hand of Malcolm Bain, who told  us about the legal limits of data cloud services, and what can we expect and demand and to what extent, from the cloud service provider, we have to offer a minimum service level. It is undoubtely surprising to learn that some of the contracts we accept on very well known services (like web mail or hosting) are, in fact, ilegal.

The battle of free software is ended, now we have to battle on freedom and privacy of data.

It has also been delightful to see the debate between some speakers, like when Javier de la Torre (Cartodbtalked against OGC or the government repositories of data. Undoubtfully a theme which, although I don’t fully agree, needs some discussion and improvement. Should government repositories of data become more useer friendly or should they remain just as repository of data?

On thursday evening, Geocat had a double session between speeches very focused on metadata and the importance of governmment spatial portals. We first focused mostly on GeoNetwork and Bridge, talking also about our future GeoCat Live. and, just before ending  the day, a workshop of 30 minutes to present GeoNetwork, which has followed by the first national meeting of geoinquietos.

It is hard to summarize in only one article all the huge dimension reached on this conference. A lot of optimism, willing to keep working and, most of all, the certainty that we are on the right path, helping free software and free data, focusing on the social part of our work. On Friday, almost to finish, Javier Sánchez oriened the end of the conference to this theme, talking about social companies, who have not only an economic result but also a social one.

In the end the Jornadas SIG Libre Girona are undoubtely the GIS Event (capital letters) to which every spanish speaker should take into account to be in contact with latest GIS news.

Original article here.