Data Mining and Data Warehouse Essay

ABSTRACT entropy archeological site, the extraction of hidden yellive information from large selective informationbases, is a powerful in the raw technology with great potential to help companies focus on the roughly grave information in their entropy wargon houses. Data digging tools predict future trim backs and behaviors, allowing origines to make proactive, knowledge- driven closings systems. Data storage storage wargonhouse is a computer system designed to give furrow finish-makers instant entrance fee to information. The w behouse copies its selective information from existing systems like order entry, general ledger, and pitying re man-made lakes and stores it for use by executives rather than programmers. Data warehouse users use redundant software that enables them to create and access information when they need it, as hostile to a reporting schedule defined by the information systems (IS) department. This piece of music describes the signification of in formation warehouse and info mine basic architecture of info wareho using and entropy minelaying, functions and working of entropy mining. It in addition presents data mining from data warehouseINTRODUCTIONModern organizations are chthonian enormous pressure with late(a) development of the technology. Clearly we need a rapid access to all kinds of information. To assist this we need to consider the erstwhile(prenominal) and to identify relevant trend analysis. So to perform any trend analysis we must let a database. In roughly organizations you willing find really large databases in operation for normal perfunctory performances. These causas of databases are known as running(a) databases in to the highest degree cases they have not been design to store historical data or to reply to queries but simply to swan all the exertions for day to day transactions.The endorsement type of database found in organizations is the data warehouse. This is designed for str ategic decision support and is largely built up from the databases that make up the operational database. The basic characteristic of a data warehouse is that it contains vast follow of data which stand mean billions of records. Smaller, local data warehouse are called data marts. A data warehouse is designed especially for decision support queries then further data that is needed for decision support is extracted from the operational data and stored in the data warehouse along with the metre when it was retrieved from operational databases.DEFINITIONDATA WAREHOUSINGA data warehouse is a subject-oriented, integrated, time-variant and non-volatile collection of data in support of managements decision qualification process. Subject-Oriented A data warehouse female genital organ be utilise to meditate a particular subject area. For caseful, sales cornerstone be a particular subject. Integrated A data warehouse integrates data from multiple data sources. For example, source A and source B may have different gists of identifying a product, but in a data warehouse, there will be only a individual way of identifying a product.Time-Variant Historical data is kept in a data warehouse. For example, one can retrieve data from 3 months, 6 months, 12 months, or even older data from a data warehouse. This contrasts with a transactions system, where often only the most recent data is kept. For example, a transaction system may hold the most recent address of a client, where a data warehouse can hold all addresses associated with a customer. Non-volatile Once data is in the data warehouse, it will not change. So, historical data in a data warehouse should never be altered.The chase are the typical move involved in the data warehousing project cycle.* Requirement convention* Physical Environment Setup* Data Modeling* ETL* OLAP Cube public figure* Front End Development* Report Development* Performance tune* Query Optimization* Quality Assurance* Rolling com e out of the closet to take* Production Maintenance* Incremental EnhancementsBenefits of a data warehouseA data warehouse maintains a copy of information from the source transaction systems. This architectural intricateity provides the opportunity to * Maintain data history, even if the source transaction systems do not. * Integrate data from multiple source systems, enabling a central view across the enterprise. This benefit is always valuable, but particularly so when the organization has grown by merger. * Improve data quality, by providing consistent codes and descriptions, flagging or even fixing fully grown data. * put in the organizations information consistently.* Provide a single common data model for all data of interest regardless of the datas source. * Restructure the data so that it makes sense to the pipeline users. * Restructure the data so that it delivers excellent query performance, even for complex analytic queries, without impacting the operational systems . * Add value to operational business drills, notably customer dealingship management (CRM) systems.Data dig (DM)Data mining, also known as knowledge find, refers to computer-assisted tools and techniques for sifting by dint of and analyzing these vast data stores in order to find trends, patterns, and cor transaction that can guide decision making and increase understanding. Data mining covers a wide variety of uses, from analyzing customer purchases to discovering galaxies.In essence, data mining is the equivalent of conclusion gold nuggets in a mountain of data. The monumental task of finding hidden gold depends firmly upon the power of computers The purpose of DM is to analyze and understand past trends and predict future trends.By predicting future trends, business organizations can better horizon their products and services for financial gain. Nonprofit organizations have also achieved significant benefits from data mining, such(prenominal) as in the area of scientifi c progress. The concept of data mining is simple yet powerful. The simplicity of the concept is deceiving, however. Traditional methods of analyzing data, involving query-and-report approaches, cannot do by tasks of such magnitude and complexity. Data mining consists of five major elements* Extract, transform, and send transaction data onto the data warehouse system. * Store and manage the data in a multidimensional database system. * Provide data access to business analysts and information technology professionals. * Analyze the data by application software.* Present the data in a reusable format, such as a graph or table.Data mining services can be used for the following functions * Research and surveys Data mining can be used for product research, surveys, market research and analysis. Information can be gathered that is quite useful in driving new merchandise campaigns and promotions. * Information collection Through the web scraping process it is thinkable to collect infor mation regarding investors, investments and funds by scraping through with(p) link up websites and databases. * Customer opinions Customer views and suggestions play an important role in the way a company absorbs. The information can be readily be found on forums, blogs and other resources where customers freely provide their views. * Data see Data collected and stored will be not be important unless scanned. Scanning is important to identify patterns and similarities contained in the data.* Extraction of information This is the touch of identifying the useful patterns in data that can be used in decision making process. This is so because decision making must be establish on sound information and facts. * Pre-processing of data Usually the data collected is stored in the data warehouse. This data needs to be pre-processed.by pre-processing it means some data that may be deemed unimportant may therefore re removed manually be data mining experts.* vane data meshing data usua lly poses more challenges in mining. This is so because of its nature. For instance, web data can be deemed as dynamic meaning it keeps changing from time to time. on that pointfore it means the process of data mining should be repeated in regular intervals. * Competitor analysis There is a need to understand how your competitors are fairing on in the business market. You need to know both their weaknesses and strengths. Their methods of marketing and distribution can be mined. How they reduce their overall costs is also quite important.* Online research The meshing is highly regarded for its huge information. It is evident that it is the largest source of information. It is possible to gather a lot of information regarding different companies, customers and your business clients. It is possible to detect tricks through online means. * News Nowadays with almost all major newspapers and news sources invoice their news online it is possible to gather information regarding trends and other critical areas. In this way, it is possible to be in the better position of competing in the market. * update data This is quite important. Data collected will be uneffective unless it is updated. This is to ensure that the information is relevant so as to make decisions from it.How does data mining work?While large-scale information technology has been evolving reissue transaction and analytical systems, data mining provides the link between the two. Data mining software analyzes relationships and patterns in stored transaction data based on open-ended user queries. Several types of analytical software are obtainable statistical, machine learning, and neural earningss. Generally, any of four types of relationships are sought * Classes Stored data is used to locate data in predetermined groups. For example, a eating house chain could mine customer purchase data to determine when customers predict and what they typically order.This information could be used to increas e traffic by having daily specials. * Clusters Data items are grouped according to logical relationships or consumerpreferences. For example, data can be mined to identify market segments or consumer affinities. * Associations Data can be mined to identify associations. The beer-diaper example is an example of associative mining. * Sequential patterns Data is mined to anticipate behavior patterns and trends. For example, an open-air(prenominal) equipment retailer could predict the likelihood of a backpack being purchased based on a consumers purchase of sleeping bags and hiking shoes.Industries/fields where data mining is flowingly applied are as follows 1. Data tap in the Banking Sector Worldwide, banking sector is ahead of many other industries in using mining techniques for their vast customer database. Although banks have employed statistical analysis tools with some success for several years, previously unseen patterns of customer behavior are now coming into clear focus wit h the care of new data mining tools. These statistical tools and even the OLAP find out the answers, but more advanced data mining tools provide sixth sense to the answer. Some of the applications of data mining in this industry are (i)Predict customer reaction to the change of interest rates (ii)Identify customers who will be most receptive to new product offers (iii)Identify loyal customers(iv) Pin point which clients are at the highest risk for defaulting on a loan (v)Find out persons or groups who will opt for each type of loan in the following year (vi)Detect fraudulent activities in credit card transactions (vii)Predict clients who are likely to change their credit card affiliation in the future(a) quarter (viii)Determine customer preference of the different modes of transaction namely through teller or through credit cards, etc.2. Data minelaying in the amends SectorInsurance companies can benefit from young data mining methodologies, which help companies to reduce cost s, increase profits, retain current customers, acquire new customers, and develop new products .This can be done through (1)Evaluating the risk of the summations being insured taking into account the characteristics of the asset as well as the owner of the asset. (2)Formulating Statistical Modeling of Insurance Risks(3)Using the Joint Poisson/Log-Normal Model of mining to optimize insurance policies (4)And finally finding the actuarial Credibility of the risk groups among insurers3. Data Mining in telecommunicationAs on this date, every activity in telecommunication has used data mining technique.(1)Analysis of telecom service purchases(2)Prediction of telephone calling patterns(3)Management of resources and net income traffic(4)Automation of network management and maintenance using artificial news program to diagnose and repair network transmission problems, etc4. Data Mining in Fraud DetectionData dredging has found wide and useful application in various fraud sleuthing proces ses like (1)Credit card fraud sensing using a combined parallel approach (2)Fraud detection in the voters list using neural networks in combination with symbolic and analog data mining. (3)Fraud detection in passport applications by invention a specific online learning diagnostic system. (4)Rule and analog based detection of false medical claims and so on.An Architecture for Data MiningTo silk hat apply these advanced techniques, they must be fully integrated with a data warehouse as well as flexible interactive business analysis tools. Many data mining tools currently operate outside of the warehouse, requiring extra steps for extracting, importing, and analyzing the data. Furthermore, when new insights require operational implementation, desegregation with the warehouse simplifies the application of results from data mining. The resulting analytic data warehouse can be applied to improve business processes throughout the organization, in areas such as promotional campaign man agement, fraud detection, new product rollout, and so on. Figure 1 illustrates an architecture for advanced analysis in a large data warehouse.Figure 2 Integrated Data Mining ArchitectureFROM DATA WAREHOUSE TO DATA MININGDM is a set of methods for data analysis, created with the aim to find out specific dependence, relations and rules related to data and making them out in the new, higher-level quality information. As distinguished from the data warehouse, which has unique data approach, DM gives results that show relations and interdependence of data. Mentioned dependences are mostly based on various numerical and statistic relations.Figure 3 Process of knowledge data discoveryEMERGING TRENDS IN DATA MINING sack up mining is the application of data mining techniques to discover patterns from the net. According to analysis targets, web mining can be divided into three different types, which are Web usage mining, Web content mining and Web structure mining. Web usage miningWeb usa ge mining is the process of extracting useful information from server logs i.e. users history. Web usage mining is the process of finding out what users are looking for on Internet. Some users might be looking at only textual data, whereas some others might be interested in multimedia data. Web structure miningWeb structure mining is the process of using graph theory to analyze the node and connection structure of a web site. According to the type of web structural data, web structure mining can be divided into two kinds 1. Extracting patterns from hyperlinks in the web a hyperlink is a structural component that connects the web knave to a different location. 2. Mining the document structure analysis of the tree-like structure of page structures to describe HTML or XML tag usage. Web content miningWeb content mining is the mining, extraction and integration of useful data, information and knowledge from Web page contents. Data Stream Mining is the process of extracting knowledge structures from continuous, rapid data records. A data stream is an ordered sequence of instances that in many applications of data stream mining can be read only once or a small number of times using limited computing and storage capabilities. Examples of data streams include computer network traffic, phone conversations, ATM transactions, web searches, and sensor data.