Groovy for crawlling website meta tags

How to quickly get html meta tags of a page ?

Here you can grab a short example of how to crawl html meta tags of a page.

import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;

def url = "http://wordpress.com";

def document = Jsoup.connect(url).get();
def metaTags = document.getElementsByTag("meta");
for (Element metaTag: metaTags) {
    def tagName = metaTag.attr("name")
    def tagProperty = metaTag.attr("property")
    def tagContent = metaTag.attr("content")
    println String.format("Name: %s, property : %s, content : %s", tagName, tagProperty, tagContent);
}

REST services stubbing example using springboot and wiremock

Integration testing of microservices

The main goal of Service Oriented Architecture is to define services that focus on delivering small, compact and most complete piece of business logic.

To deliver complete user features, many services need to communicate with each other. That communication is often implemented as simple REST requests.

The example

I’ve prepared a simple repository with working example of a simple rest service implemented using springboot and integration tested using wiremock.

Please, checkout the repo : https://github.com/youngeagle87/wiremock-example

You can run the tests using command :

for windows :

gradlew test

for any other proper commandline

./gradlew test

Mock REST services using Restito Examples

Integration testing

I have been developing rest microservices for years now, and I always had best results by focusing on the integration testing of the services, instead of covering almost 100% of code by unit tests.

I usually use one of two tools : Wiremock and Restito.

The choice between them is not obvious. It’s rather a team decision or preference. Usually the tool that have been used more recently wins, without massive superiority over the other option.

Restito

It is a framework that can be used for stubbing http responses, simulating other microservice responses. The power of such stubs is that you can test your application against several malfunction of services that your application depends on.

Example 1. The base, extendable integration test specification

The code below shows the usage of restito StubServer for stubbing a remote service. Toolset : Spock and Groovy.


class IntegrationSpecification extends Specification {

	private static final Integer port = 9090;

	protected StubServer server;

	@Shared
	ConfigurableApplicationContext context

	@Before
	public void start() {
		server = new StubServer(port).run();
	}

	@After
	public void stop() {
		server.stop();
	}

	void setupSpec() {
		Future future = Executors
			.newSingleThreadExecutor().submit(
			new Callable() {
				@Override
				public ConfigurableApplicationContext call() throws Exception {
					return (ConfigurableApplicationContext) SpringApplication
							.run(YourMainApplication.class)
				}
			})
		context = future.get(60, TimeUnit.SECONDS)
	}

	void cleanupSpec() {
		if (context != null) {
			context.close()
		}
	}
}

In the code above, there is a setupSpec() method. It is not obligatory for you to have it like this. It is an example of how your Spring application can be started by Spock tests.

Example 2. The test implementation.

@IntegrationTest
@SpringApplicationConfiguration(CogniTwitterApplication.class)
@ActiveProfiles("test")
class SearchEndpointTest extends IntegrationSpecification {

    private om = new ObjectMapper()

    def "get search params should return filled values when set"() {
        given:
        def restTemplate = new RestTemplate();
        def oauthResponse = new AccessGrant("exampleAccessToken");
        def searchResponse = [new TweetDto(1,"test",new UserDto("testuser",null),null,"2014-01-01 00:00:00")] as List

        //first Restito Stub
        whenHttp(server).
                match(get("/oauth2/token")).
                then(
                    status(HttpStatus.OK_200),
                    stringContent(om.writeValueAsString(oauthResponse))
                );

        //second Restito Stub
        whenHttp(server).
                match(get("/1.1/search/tweets.json?q=*")).
                then(
                    status(HttpStatus.OK_200),
                    stringContent(om.writeValueAsString(searchResponse))
                );

        HttpHeaders headers = prepareSelectedSearchParamsSession(restTemplate)
        when:

        def response2 = restTemplate.exchange(
                "http://localhost:8080/search",
                HttpMethod.GET,
                prepareEntityWithHeaders(headers),
                TweetDto[].class);

        then:
        response2.statusCode.value() == HttpStatus.OK_200.statusCode
    }

    private HttpHeaders prepareSelectedSearchParamsSession(RestTemplate restTemplate) {
        def headers = prepareHeaders()
        def entity = new HttpEntity("{\"keywords\":\"spring\",\"channel\":null}", headers)

        def response = restTemplate.exchange("http://localhost:8080/searchParams", HttpMethod.POST, entity, Void.class);
        def session = response.getHeaders().get("Set-Cookie").get(0).split(";")[0];

        headers.add(COOKIE, session);
        headers
    }

    private HttpEntity prepareEntityWithHeaders(headers) {
        def entity = new HttpEntity(null, headers)
        entity
    }

    private HttpHeaders prepareHeaders() {
        def preparedHeaders = new HttpHeaders();
        preparedHeaders.add(CONTENT_TYPE, APPLICATION_JSON_VALUE);
        preparedHeaders
    }
}

Running Spring boot application as a service (linux)

You can easily make your application run as a service on your hosts.
If you have few machines, you can use simple init.d service

To make your built application start as a service, all you need to do is :

sudo link -s /path/to/your/app/yourapp.jar /etc/init.d/yourappservice

And directly after that you can start the yourappservice as you have the service configured

./etc/init.d/yourappservice start

That’t it!

On more complicated environments (many instances, or cloud env) you should probably use supervisor to control your services.
You can check it out here : http://supervisord.org/running.html#adding-a-program

Creating your first Composer/Packagist package | Junior Grossi

Junior Grossi wrote quick tutorial about creating composer packages

“Hi everybody! Today I’ll write about how you can contribute with PHP community creating packages (or updating your’s) using Composer and Packagist. First, if you’re a PHP developer and don’t know yet what is Composer, take a look on the post Why you should use Composer and how to start using it to get more information about.”

Check out the source : Creating your first Composer/Packagist package | Junior Grossi

Just Enough Regular Expressions for Cucumber – Agile For All

Jon Archer wrote about how Cucumber makes knowledge of regular expressions important. He’s right: Regular expressions are the key to Cucumber’s flexibility. Well-crafted regular expressions let you reuse step definitions, avoiding duplication and keeping your tests maintainable. But even experienced developers … ( Read More on source )

Source: Just Enough Regular Expressions for Cucumber – Agile For All

Cucumber statement definition parameters : 1st, 2nd, 3rd, 4th

Sometimes there is need to handle one of many ordered items on page, we can use integer as identifier, but it looks unnatural, for example :

User named “John” is author of 1 article

The number is misleading : it can be read as one, but we would like it to be read like “first”.

Below there is a Java example, where you can use both patterns (1st or 1).

 @Then("^User named \"(.*)\" is author of \"(\\d+)(?:st|nd|rd|th)\" article$")
public void UserNamedIsAuthorOfNthArticle(String name, int nth) {
    boolean isAuthor = testPage.getNthArticle(nth).getAuthor().equals(name);
    assertTrue("User named " + name + " is author of " + nth + " article", isAuthor);
}