Groovy for crawlling website meta tags

How to quickly get html meta tags of a page ?

Here you can grab a short example of how to crawl html meta tags of a page.

import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;

def url = "http://wordpress.com";

def document = Jsoup.connect(url).get();
def metaTags = document.getElementsByTag("meta");
for (Element metaTag: metaTags) {
    def tagName = metaTag.attr("name")
    def tagProperty = metaTag.attr("property")
    def tagContent = metaTag.attr("content")
    println String.format("Name: %s, property : %s, content : %s", tagName, tagProperty, tagContent);
}

Mock REST services using Restito Examples

Integration testing

I have been developing rest microservices for years now, and I always had best results by focusing on the integration testing of the services, instead of covering almost 100% of code by unit tests.

I usually use one of two tools : Wiremock and Restito.

The choice between them is not obvious. It’s rather a team decision or preference. Usually the tool that have been used more recently wins, without massive superiority over the other option.

Restito

It is a framework that can be used for stubbing http responses, simulating other microservice responses. The power of such stubs is that you can test your application against several malfunction of services that your application depends on.

Example 1. The base, extendable integration test specification

The code below shows the usage of restito StubServer for stubbing a remote service. Toolset : Spock and Groovy.


class IntegrationSpecification extends Specification {

	private static final Integer port = 9090;

	protected StubServer server;

	@Shared
	ConfigurableApplicationContext context

	@Before
	public void start() {
		server = new StubServer(port).run();
	}

	@After
	public void stop() {
		server.stop();
	}

	void setupSpec() {
		Future future = Executors
			.newSingleThreadExecutor().submit(
			new Callable() {
				@Override
				public ConfigurableApplicationContext call() throws Exception {
					return (ConfigurableApplicationContext) SpringApplication
							.run(YourMainApplication.class)
				}
			})
		context = future.get(60, TimeUnit.SECONDS)
	}

	void cleanupSpec() {
		if (context != null) {
			context.close()
		}
	}
}

In the code above, there is a setupSpec() method. It is not obligatory for you to have it like this. It is an example of how your Spring application can be started by Spock tests.

Example 2. The test implementation.

@IntegrationTest
@SpringApplicationConfiguration(CogniTwitterApplication.class)
@ActiveProfiles("test")
class SearchEndpointTest extends IntegrationSpecification {

    private om = new ObjectMapper()

    def "get search params should return filled values when set"() {
        given:
        def restTemplate = new RestTemplate();
        def oauthResponse = new AccessGrant("exampleAccessToken");
        def searchResponse = [new TweetDto(1,"test",new UserDto("testuser",null),null,"2014-01-01 00:00:00")] as List

        //first Restito Stub
        whenHttp(server).
                match(get("/oauth2/token")).
                then(
                    status(HttpStatus.OK_200),
                    stringContent(om.writeValueAsString(oauthResponse))
                );

        //second Restito Stub
        whenHttp(server).
                match(get("/1.1/search/tweets.json?q=*")).
                then(
                    status(HttpStatus.OK_200),
                    stringContent(om.writeValueAsString(searchResponse))
                );

        HttpHeaders headers = prepareSelectedSearchParamsSession(restTemplate)
        when:

        def response2 = restTemplate.exchange(
                "http://localhost:8080/search",
                HttpMethod.GET,
                prepareEntityWithHeaders(headers),
                TweetDto[].class);

        then:
        response2.statusCode.value() == HttpStatus.OK_200.statusCode
    }

    private HttpHeaders prepareSelectedSearchParamsSession(RestTemplate restTemplate) {
        def headers = prepareHeaders()
        def entity = new HttpEntity("{\"keywords\":\"spring\",\"channel\":null}", headers)

        def response = restTemplate.exchange("http://localhost:8080/searchParams", HttpMethod.POST, entity, Void.class);
        def session = response.getHeaders().get("Set-Cookie").get(0).split(";")[0];

        headers.add(COOKIE, session);
        headers
    }

    private HttpEntity prepareEntityWithHeaders(headers) {
        def entity = new HttpEntity(null, headers)
        entity
    }

    private HttpHeaders prepareHeaders() {
        def preparedHeaders = new HttpHeaders();
        preparedHeaders.add(CONTENT_TYPE, APPLICATION_JSON_VALUE);
        preparedHeaders
    }
}