Lacisoft's » Programming

A little crawling experiment

lacisoft — Wed, 21 Dec 2011 08:42:33 +0000

Almost a year ago i wrote the article about the TYPO3 market share statistics i’m preparing to make a new statistic in the early days of 2012. Having too much time on my hands last days i thought why not to write my own crawler which searches for TYPO3 sites ?

I know this is quite a difficult task. Actually it is not that difficult to write the crawler, its much more difficult to scale after the number of domains start to grow in the database. But i like challenges.

So i started working on it and after a few hours a basic crawler was born. What it does is that it scans a website and tries to identify it if it is a TYPO3 website (based on the TYPO3 header comment). Then it extracts all distinct domains from the links and goes to the next domain.

The early data is available here: http://crawler.lacisoft.com/ The crawler is not very speedy because i limited its speed (i don’t want to kill my VPS). Also because of this i limited its scope to .ro domains (domains from Romania).

Another issue is that for now it counts a domain twice if it has both www and non-www version. Technically i could solve it to count only one but i’m not convinced that this would be the right step as theoretically www is a subdomain of the main domain and it is possible that the www and non-www versions house different websites. I will think more about it.

For now this is only an experiment, let’s see how it works out.

Later edit: The crawler is still scanning, the data is not final!

TYPO3 Usergroup Romania Meetup 2011–schedule

Learn to say “No”

lacisoft — Wed, 09 Jun 2010 05:51:58 +0000

All too often i receive emails or inquiries from people on how much would some website that they imagined would cost and how long it would take for me to implement it. Because this blog lately become more popular then ever before, i started to receive even more of these inquiries . One might say that very good for me, that in these harsh times i receive work offers. But many of these offers will not get any reply from me or get a negative reply. Why is that ?

This is because i’m not willing to take work that will eventually be turning into bad reputation for me or the client. There are two situations in which this might occur:

1) Stealing websites

For example i often get inquires like this:

“Hi. My name is xxxxxx and i’ve seen you build websites with TYPO3. I would like you to build me a website for a client of mine. I’ve seen this site yyyyyyyy and i would like you to copy this site and change the logo and the texts in it. How much would you charge for this ?”

I rarely reply to this kind of emails. That is because i don’t want to say to someone that he is a thief. Copying websites its just plain stealing if you don’t have the owners approval. First time when i said this the client said: ”that website is in Australia, who will check it anyway ?”. No comment.

2) The “i want Facebook in like a week” approach

The other category of inquiries that i turn down will come with this type of emails:

“Hi. My name is xxxxxx and i want to make a website just like Facebook but better, i want to integrate it with Twitter and also a search like Google would be nice on it. My budget would be 1200 EUROS and i need it in two weeks. Would you be interested ?”

No. Most of the times i’m not interested. The guys at Facebook worked on it for a few years and Twitter is still having issues under load , Google isn’t matched yet as performance (after being tweaked for ten years) but someone wants to build all that in two weeks. Ok most of the times its not Facebook, Twitter or Google they want but the situation is the same. They want space rockets built out of thin air.

Ok, i might take the project and work on it for two weeks and get some payment for it. But in the end the client wouldn’t be happy with the fact that the product he could get in two weeks is not the product he dreamed of. I would get bad reputation, he would search for someone else.

So when one of the two situations above occurs i learned to say no to these projects. Because its more important to have good projects that you can be proud of working on then many runaway projects that never get completed, or get completed in a manner that will destroy your reputation.

Also when someone making or requiring an estimation or a quote should take into account that there are 3 types of work:

quality work
fast work
cheap work

Quality work will never be fast and cheap. It will be either fast and expensive (more developer resources) or cheap but slow.

Fast work will never be cheap and quality work. It will be either cheap and miserable or it will be quality work but it will be expensive. Very expensive.

Cheap work will never be quality work and fast work in the same time. It will be either quality work and slow work or it will be fast work but it won’t be of any acceptable quality.

So if you have to choose always choose two from these three: quality, fast, cheap. Its not possible to choose all three of them.

How Google contextual ads learn your preferences over time

A lesson to be learned in software development

lacisoft — Tue, 22 Dec 2009 17:12:11 +0000

I’ve just read a great and insightful article on Wired about the development of a game that was never finished and it was canceled after 12 years of development. The game is Duke Nukem Forever and the article is called: Learn to let go: How success killed Duke Nukem

The situation from the article happens often when there is no direction, no milestones, no plan. If the plan is that there is no plan, then the project will certainly fail.

Interest for PHP is declining ?

lacisoft — Wed, 25 Mar 2009 07:53:21 +0000

Google Insights for Search is an interesting tool, it allows you to take a look at what people are serching for in different parts of the world and how is the volume of search evolving for a keyword in a specific part of the world or worldwide.

I played a bit with Google Insights and after a while i tought i check how PHP search volume is evolving, after all i work with PHP and i wanted to see what is graph for the popularity of this language. I must say i was shocked. The interest over time from 2004 to 2008 is declining, rapidly i could say. But don’t take my word for it, check in Google Insights, or here is the picture (click on it for better quality):

I don’t understand this decline as interest for many PHP applications are increasing sharply in this period, you could check Google Insights for WordPress , Joomla or other PHP based applications.

The only thing i could think about is the proliferation of these already built web apps, that are downloaded and installed, in many cases the users get an already functional application so they don’t have to bother do understand or learn about PHP. Back in the times when you needed a CMS or a website you had to build it yourself using PHP or at least you needed to have some PHP knowledge to install it. Now with the advanced click click next next installers users don’t need to know anything about PHP.

What is your opinion on this ?

Gmail is down! How to check if it is really down or its just your connection

How to scrape webpages using PHP and XPath

lacisoft — Thu, 19 Feb 2009 16:55:02 +0000

If you need to extract information from remote webpages you can read this article about how to do it with PHP and XPath. The autor explains how extracting information can be very easy and it can be done in no time (less then 5 minutes) with the mentioned tools and without the use of regular expressions.

No related posts.

How to generate PDF files dynamically using PHP and Haru

lacisoft — Thu, 20 Nov 2008 11:47:06 +0000

An excellent tutorial on how to create PDF files dynamically using PHP and Haru written by Will Sinclair. And here is a quote from the article that grabbed my attention

“When it comes to working with different file formats, PHP is like a Swiss Army knife: it has a tool to handle almost anything you can throw at it. JPEG images, XML files, Word documents, ZIP archives…you name it, and there’s a PHP extension to handle it!”

Tips for improving your PHP programming skills

lacisoft — Tue, 18 Nov 2008 19:43:44 +0000

Smashing Magazine published an article with 10 tips to improve your PHP programming skills. They claim to be advanced tips but i tend to disagree. Every PHP programmer with 1-2 years of experience, somewhere around mid-level should know these things. But nevertheless the article is worth reading, if you did not know those things you may remember them again.

A comparation between Typo3 and Drupal

lacisoft — Mon, 03 Nov 2008 07:55:11 +0000

Working in web development often requires the use of different CMS packages that have certain advantages or disadvantages. At work Typo3 and Drupal are the most often used content management systems. I will make a short comparation from a web developers perspective on these two packages.

Learning to work with them

I first started to work with Typo3 and i must say that Typo3 is hard to learn. The documentation is scarce and very badly structured. It is a nightmare for every beginner. But once you start to learn the inner workings of this package you become more and more amazed by the power that is under the hood of this CMS.

Drupal is somewhat luckier having more documentation and more books written about it. Learning to work and develop sites using it is easier somewhat.

Working with forms

Drupal has a very ingenious system for forms. I was amazed when i learned to use it. You just need to create a configuration array for forms and call them from your module. Drupal will take care of generating the forms and submit it and validate it according to your preferences. And the AHAH form property is just killer. It will make Ajax calls and update the forms accordingly.

Typo3 has some plugins for forms and frontend editing but they aren’t there yet where it should be, it’s just a pain sometimes to use multilevel forms.

Templates

Drupal comes with a default template and you can start using that right away. Typo3 comes empty, you will need to create a template and code it into your project. You would think that Drupal is better from this point of view. But this is not true. Drupal’s templates aren’t really to good. That’s because in Drupal’s templates PHP code is mixed with HTML, the two aren’t separated as clearly as it should be. If you work with a designer who can make HTML code for you but he has no PHP knowledge, he will have a hard time working with Drupal templates.

In Typo3 HTML templates are what the name says, pure HTML with markers for different content elements. So creating a template is pretty straightforward even for people with no coding experience.

Customization

Drupal is well suited for community sites and often you just need to enable some modules bundled with the package like blog, forums, poll, faq management and modify the templates. But the problem is that often these modules provide basic functionality and they just aren’t what you want in your app.

In contrast Typo3 is much more complex and you can customize everything. You don’t have bundled plugins like forum or blog but you can download them for free from the extension repository. And most often they are more powerful and much more customizable then those included with Drupal.

Drupal

Pros: documentation, forms, out of the box functionality
Cons: templating

Typo3

Pros: templating, highly customizable
Cons: hard to learn

Conclusion

Both CMS packages have their pros and cons, it depends on the developer to choose the right CMS for the job required to be done. Because for some projects is more suited to use Drupal and for other projects is better to use Typo3. But the conclusion is that you need to learn them both, you might need one or another.

PHP support may be on the way in Google App Engine

lacisoft — Thu, 30 Oct 2008 08:38:24 +0000

Rumor has it that Google is adding support for a new runtime language for Google App Engine. If you take a closer look at their roadmap you can see that for sure another runtime language will be added. But which one will be ? Java or PHP ? According to some reports, Google said that they cannot yet say the name of the new runtime language but they indicated that is one being demanded by the community. If you follow what languages were requested you can see that Java is in the lead, followed by PHP. If you want to express your vote to PHP just go here and star that thread. You don’t need to post anything since posting will send an email to everyone. Let’s hope PHP will be the one choosen!

Controversed namespace separator for PHP namespaces

lacisoft — Tue, 28 Oct 2008 07:47:23 +0000

The PHP development team decided to use the backslash (\) as namespace separator in future releases of PHP (up from 5.3) The decision wasn’t very welcomed by the community and numerous people already started basing PHP again. i feel that some of the people involved are beeing very single minded on this. I think time will tell who was right and who wasn’t. I personally would liked to be :: instead of \ but i will live with this. This shouldn’t be a motive for anyone to spread the FUD (Fear Uncertainty Doubt) around about PHP.

No related posts.